Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
# Architecture and domain contracts

D430 lets the curve page load only the exact definition dependencies of the
selected curve. Pricing, frame validation and publication authorization use
the original inputs and policies. See [the decision](decisions.md#d430--read-exact-curve-definition-dependencies).

D429 separates first-use curve catalog hydration from its bounded price result
and streams owned read bytes and calendar batch input. Existing projections and
authorization remain authoritative; see [the decision](decisions.md#d429--separate-curve-catalog-hydration-and-bound-read-serialization)
and [qualified workloads](verification.md#curve-hydration-and-read-memory-d429).

D427 adds opt-in personal [assistant history search](assistant-workspace.md#personal-chat-history-search-d427)
and retains walkthrough guidance in conversation history. Search uses current
user/agent authority and returns prior interactions with their original context;
Expand Down
32 changes: 28 additions & 4 deletions docs/async-reads.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,10 +94,15 @@ before admission; the client renews a drained channel before
submitting further requests. Request and serialized-result caps are 32 KiB and
16 MiB. A channel lasts five minutes and uses periodic authorization checks and
heartbeat writes. Work cancellation is cooperative: synchronous graph evaluation
cannot be forcibly preempted by a cancellation token. A serialized-result cap is
not a peak-memory guarantee. Publication evidence is bounded to 512 source or
reference/calendar coordinates, 4,096 original book identities and 4,096 retained
instrument authority proofs.
cannot be forcibly preempted by a cancellation token. The result is serialized
once into an owned UTF-8 buffer, with its byte budget checked at writer commits
before publication, including DOM values. Publication writes that buffer in 16
KiB chunks inside the existing authorization callback; one write deadline covers
the complete event and its final flush. The existing 64-level result nesting
limit is preserved inside the additional envelope level. A serialized-result cap
is not a peak-memory guarantee. Publication evidence is bounded to 512 source or
reference/calendar coordinates, 4,096 original book identities and 4,096
retained instrument authority proofs.

Changing a selection rejects the old consumer immediately. A submitted
cancellation keeps its execution slot until a terminal server response or an
Expand All @@ -110,6 +115,25 @@ an aborted HTTP submission that never reached admission to release its slot.
Disconnect rejects pending reads. A subsequent explicit request establishes a new channel;
there is no silent replay or durable delivery guarantee.

The curve page hydrates its catalog through the existing ordinary
`workspace/catalog` read before its first price read. It validates both results
against the same captured selection and installs them together. Later price
reads reuse the scoped catalog. This keeps a large catalog outside the price
result's byte budget; it adds a first-use HTTP/database read. The explicit
`includeWorkspace` API option retains its combined-result contract and byte cap.

The page uses `workspace/catalog?frameDetail=summary` to omit per-frame input
IDs and requests `sourceScope: "curve_lineage_v1"` with `includeSources: true`
to receive only the point lineage anchors and their visible fact versions.
The full frame remains the authority and correction-admission witness.
The page schedules the curve's advisory close strip after the prices paint.

For both ordinary HTTP and `curve.evaluate.v1`, the page also sends
`definitionScope: "curve_dependencies_v1"`. This selector requires the
curve-lineage projection and a separate catalog read. The server reads
the selected curve's declared dependency closure under the existing caller
and exact frame authority. Omitting the selector uses the full definition read.

Financial commands retain their existing request identities, transaction ordering
and uncertain-outcome recovery. Retained Risk jobs keep their existing durable
admission and report contracts. Progressive trade rows retain their dedicated
Expand Down
59 changes: 59 additions & 0 deletions docs/decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -6651,6 +6651,65 @@ hydration advantage for the tested Protobuf candidates over gzip JSON; see the
measurements do not establish a production latency guarantee. Broader transport,
projection-size or deployment changes require separate qualification.


## D429 — Separate curve catalog hydration and bound read serialization

The curve page reads its catalog through the existing `workspace/catalog` HTTP
operation, then requests prices without embedding that catalog in the SSE result.
Both reads retain the captured profile, generation, selected frame and source
revision checks; the page installs their data only while that selection remains
current. A hydrated catalog is reused within its existing scope. First hydration
adds one ordinary HTTP read and its database work. Explicit API callers may still
request the existing combined projection.

The page requests frame summaries and sends an explicit curve-lineage source
projection. The evaluation still binds the same full input frame, checks its
original pins and reauthorizes before publication. The price payload includes
the anchors used by its points and visible versions of those same facts.
Correction eligibility remains tied to the full selected frame. Browser indexes
preserve exact-lookup failures. The close strip starts after the chart can paint
and a superseded strip read is cancelled. This scheduling applies to the
advisory strip when selecting a curve, while choosing a close still uses its
authorized availability read.

Keep the channel's 16 MiB serialized-result limit. Enforce actual committed JSON
bytes at the UTF-8 writer boundary, including synchronous JSON DOM converters,
and retain the successful buffer for publication. The same authorized
publication callback writes its correlated envelope and result in bounded chunks
under one write deadline. It preserves exact values, structured refusals and
cancellation; a truncated event cannot be completed as a successful response.
The existing 64-level result nesting bound is preserved inside its additional
envelope level.

Calendar pin batches read the same native function once, stream its UTF-8 payload
through a sequential database reader, and retain individual rows and denied-frame
identities. Numeric lexeme conversion, complete pin selection, reached-frame
visibility evidence and denial/error precedence keep their existing contracts.
The aggregate UTF-16 string and aggregate JSON document are unnecessary copies.
No schema, stored format, financial input or command-identity change is involved.

The logical byte limit is not a heap ceiling. Encoding reservations, evaluation,
metadata and the complete calendar observation union still consume memory.
[Qualification](verification.md#curve-hydration-and-read-memory-d429) records
separate browser, serializer, catalog and batch workloads; it does not qualify
arbitrary batch sizes, simultaneous large reads or production latency.

## D430 — Read exact curve definition dependencies

The curve page requests the versioned definition scope
`curve_dependencies_v1` with the compact curve-lineage result. The ordinary
HTTP and asynchronous operations admit the same scope. The native read follows
declared dependencies inside the bound exact definition and retains all
selected curve members, mappings, schedules and rate tenors needed by the
existing evaluator. Series and contract metadata for every original selected
pin remain available for full snapshot validation. Row security and publication
checks use the original caller, frame and input identities.

There is no cross-request definition cache. Keep the full definition read for
callers that omit the selector and for legacy supplementary sources without a
recorded exact source matrix. Retain gzip JSON and the separate catalog read.
[Verification](verification.md#curve-definition-dependency-reads-d430) qualifies
the warm price path on two owned synthetic workloads.
## D429 — Manage a registered risk worker service from Jobs

An installation binds one reviewed Swarm service and a dedicated risk-worker login to its workspace. Jobs can request zero, one or two replicas. Scaling requires both workspace permission and current installation operator registration. A separate permission grants access to bounded worker and node diagnostics. Installations choose the roles that receive these permissions.
Expand Down
14 changes: 14 additions & 0 deletions docs/plan.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,19 @@
# Delivery plan and acceptance criteria

D429 qualifies separate catalog hydration, bounded asynchronous JSON serialization
and streamed calendar batch decoding in owned synthetic workloads. Larger
concurrent workloads, first-use request-count costs and production latency remain
capacity/interaction measurements; the logical response cap is unchanged. See
[verification](verification.md#curve-hydration-and-read-memory-d429).

D430 adds an explicit dependency-scoped definition read to curve evaluation.
[Local qualification](verification.md#curve-definition-dependency-reads-d430)
covers exact response parity and warm click-to-chart time with unrelated
contracts at two market sizes. Cold hydration, production-like latency and
concurrent capacity remain open. A curve-scoped display projection and managed
display formatting in the price response remain proposed changes.


D427 adds personal chat-history search through an explicit composer checkbox and
repairs retained walkthrough narration and final-pause verification. Its bounded
contract is in [workspace assistance](assistant-workspace.md#personal-chat-history-search-d427).
Expand Down
135 changes: 135 additions & 0 deletions docs/verification.md
Original file line number Diff line number Diff line change
Expand Up @@ -5550,3 +5550,138 @@ not isolated as a comparable standalone CPU stage. The large history projection
has API parity; its unavailable-input UI states are outside the covered history
render case. These limits support retaining JSON, not a universal claim about
binary encoding or a production service-level guarantee.

## Curve hydration and read memory (D429)

Release checks on 2026-11-18 reproduce the old behavior before each memory fix:
serializing an oversized lazy result walks all 8,192 rows before rejecting it;
an oversized JSON object allocates 268,397,056 bytes before the stream limit
runs; materializing a 71,095,510-byte synthetic pin batch exhausts a 256 MiB
managed heap. The corrected writer stops the lazy result at row 2,049. Oversized
JSON object/element checks allocate 41,975,840 / 67,100,520 bytes respectively;
a definitely oversized string refuses before its encoded copy is allocated.
These are serialization allocation checks, excluding construction of their inputs.

The complete async-read checks pass exact logical UTF-8 limits at 16 MiB minus
one byte, exactly 16 MiB and one byte over, including escaped non-ASCII text.
They cover exact financial text, null/empty fields, the existing 64-level result
plus its envelope, serialization cancellation, correlated failures, partial-write cancellation,
one total event deadline, gzip/identity negotiation and ready-event flushing.
Writing a precomputed 4 MiB result adds 896 allocated bytes in its measured check;
the counting sink is preallocated and excluded. This is a publication-stage
allocation result, not the memory cost of evaluating that result.

An opt-in backend HTTP case uses actual managed native-curve metadata, 20,000
contracts, 20,000 observations, one selected contract and three sequential base
price reads. It serves a complete 17,026,511-byte catalog separately; its gzip
body is 3,928,369 bytes. Each price result preserves its complete precomputed
business projection and carries no catalog. This Release process includes both
the Kestrel server and its HTTP validation client. Its managed heap limit is
768 MiB and measured peak resident size is 759,054,336 bytes (724 MiB).

A separate input-decoding case reads 525,123 original synthetic observations
from a 252,887,126-byte UTF-8 batch and builds the native observation indexes.
It preserves every row, exact decimal text and denied identity. Under the same
768 MiB managed heap limit, measured peak resident size is 798,736,384 bytes
(762 MiB). This case includes batch decoding/indexing, not an API server,
database process or simultaneous curve evaluation. The complete batch remains
resident; larger or concurrent batches are outside this measured envelope.

Both cases run on macOS 26.6.2 arm64, 14 logical CPUs, 48 GiB host RAM,
.NET/ASP.NET Core 10.0.13 and SDK 10.0.402, using Release builds. The RSS source
is the operating system's `time -l`, rather than the runtime's unavailable
peak-working-set counter on this platform. The managed heap constraint is not
a Linux container/cgroup qualification. HTTP bytes above are compressed body
bytes over loopback, excluding headers, TLS and proxy framing. These cases do
not measure production latency, bandwidth, browser rendering or concurrent-user
capacity.

The owned actual-browser regression passes 13 checks, including a real power
curve, first hydration with over 16 MiB of injected unselected catalog metadata,
rapid curve switches, a catalog response delivered after supersession, pending
Risk reads and exact startup readiness. The metadata injection occurs after the
small fixture's HTTP response; it qualifies browser behavior rather than server
memory. The separate backend-scale case above covers server serialization.

The page projection comparison in the owned asynchronous HTTP fixture covers
four frames, 30 catalog contracts and six points of a nested power curve at one
exact close. The compact price result retains the same points, trace, selected
source and correction eligibility. It carries 31 observation rows versus 40
in the full-frame response and 51,718 versus 57,747 bytes as compact JSON
re-encoded by the test client. Frame summaries reduce the separate catalog
from 201,998 to 185,542 bytes by the same measure. Ordinary HTTP and SSE return
identical compact projections. Further owned checks cover exact rate tenors,
empty eligible delivery, a non-primary source revision and unknown projection
refusal. The payload counts exclude the response envelope and compression, and
both variants used the full market calculation input load. The owned calendar
run passed 67 browser checks. It confirms that the close strip starts after its
selected prices draw and that editing the context cancels a pending strip
response in the tested flow.

Native PostgreSQL calendar checks preserve the unchanged answer hashes, source
matrices, reached/unvisited denial precedence, original authority, current
publication permission, close replacement and profile authorization. The async
suite separately verifies profile/sign-in fences. Statement counts remain 101
for 24, 48 and 96 candidates. The macOS run adapts `ps` sampling only for the
check harness's Linux `/proc` counters; business assertions are unchanged. CPU
samples have 0.01-second resolution and calendar peak RSS is unavailable in that
sampler; these samples establish no performance claim. API/checks and Worker
Release builds pass.

The optional managed HTTP case is reproducible with:

```sh
dotnet build tests/OpenEtrm.Api.Checks -c Release
DOTNET_GCHeapHardLimit=30000000 dotnet \
tests/OpenEtrm.Api.Checks/bin/Release/net10.0/OpenEtrm.Api.Checks.dll \
--large-catalog-http
```

`DOTNET_GCHeapHardLimit` is hexadecimal here: `30000000` selects 768 MiB.
The file-input check accepts `--observation-pin-payload FILE EXPECTED_ROWS` for
an explicitly generated complete synthetic batch; generated payloads and local
operational evidence stay outside the repository. No schema migration or new
dependency is required. The first curve interaction adds an ordinary catalog
HTTP/database read; its added round-trip cost and production interaction latency
remain to be measured after an approved release.

## Curve definition dependency reads (D430)

The owned synthetic native preview compared full and dependency-scoped reads
with the same compact JSON projection, gzip configuration and browser page.
The only request difference was `definitionScope`. The target nested base
plus basis curve had 76 output points, 152 participating contracts, 152
consumed observations per close and 228 trace nodes. Four synthetic closes
were present. Scaling unrelated contracts and series raised the total native
definition from 3,031 to 12,031 contracts. Every measured HTTP result matched
its full-definition counterpart exactly.

| Contracts in definition | HTTP p50, full → scoped | HTTP sample max | SQL statements per read, median | Warm chart p50, full → scoped | Warm chart sample max |
| ---: | ---: | ---: | ---: | ---: | ---: |
| 3,031 | 102.7 → 70.3 ms | 117.9 → 82.6 ms | 98 → 47 | 131.5 → 101.2 ms | 143.9 → 111.8 ms |
| 12,031 | 191.7 → 170.8 ms | 222.9 → 188.3 ms | 98 → 47 | 233.2 → 103.4 ms | 253.8 → 107.8 ms |

Runs used macOS 26.6.2 arm64, 14 logical CPUs, 48 GiB RAM, PostgreSQL
18.7, .NET/ASP.NET Core 10.0.13, SDK 10.0.402 Release builds and headless
Chrome 157.0.0.0. Each scale used nine alternating paired HTTP reads and
eight alternating paired browser clicks on loopback. Chrome consumed the real
SSE stream and changed only the outgoing definition selector. Browser timing starts
with the click from an already hydrated six-point curve and ends after
the selected 76-point table is ready and two animation frames have elapsed.
The HTTP statement count comes from native preview runtime logs. Median
summed statement duration was 66.0 → 50.9 ms and 109.2 → 153.5 ms at the
two scales. A separate 12,031-contract HTTP run also had exact parity and
98 → 47 statements, with round-trip medians of 186.5 → 71.2 ms and summed
statement medians of 107.2 → 55.4 ms. The difference between the large-scale
HTTP runs remains unexplained. These sample maxima describe run variation
within each run. Process peak memory, cold hydration, calibrated proxy latency,
concurrent users and a 96-close dataset remain open. The dependency-scoped
reader applies to price evaluation.

The native check passed 50 HTTP/SSE assertions, including identical compact
power and rate responses under working and certified contexts. A scoped
curve blocked on the publication gate refused disclosure after its authority
expired. Separate checks passed an exact non-primary recorded source and an
expired curve with no eligible output. The 13 owned browser checks passed,
including selection supersession and late catalog delivery. All 2,244
frontend tests passed.
Loading