Feat/retire aimdb ws protocol - #201
Merged
Merged
Conversation
…sport on AimX The mapping-table design doc 036 A2 gated the protocol unification on (038 §3.9, 034 §3.10 rc10): AimX ↔ ws-protocol frame mapping, the gap decisions (wildcard subscribe via optional Event.topic, Snapshot routing via a new sub field, the explicit subscribed ack, the 6→3 error-code collapse, shared record.query/record.list result shapes), two scope corrections verified against the code (the aimdb-client demux fold was already done in PR #124; query passthrough already existed — only the result shape was missing), and the go/no-go. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r AimX (design 045)
The two features that justified the ws-protocol fork, added to AimX so
every transport inherits them:
- session::topic_match — the MQTT-style matcher moved in from
aimdb-ws-protocol (same semantics, same tests) plus is_wildcard.
- Subscription streams yield SubUpdate {topic: Option<Arc<str>>, data}
on both engines; Outbound::Event gains an optional topic and
Outbound::Snapshot the routing sub, so a wildcard subscription's
events and late-join snapshots name the record that fired.
Session::snapshot becomes snapshots() — one per covered record.
- AimxDispatch matches wildcard patterns against the registry once at
subscribe time (builder-frozen record set), merges matched streams
under one subscription id, and snapshots each match on open.
- AimxCodec learns the {"t":"subscribed"} ack (servers running
acks_subscribe:true — the WS connector) and gets a dedicated
roundtrip test suite; exact-topic frames are unchanged on the wire.
- ClientConfig::topic_routed_subs removed — it existed solely for the
retired ws wire; all subscriptions are id-routed.
- Shared result vocabulary: remote::QueryRecord {topic, payload, ts},
with_persistence's QueryHandlerFn returning {records, total} (sorted
by ts), and RecordMetadata gaining optional schema_type/entity.
- aimdb-client: AimxConnection::subscribe_with_topics yields
(Option<String>, Value) pairs; wildcard e2e test over the production
UDS server; query-shape test against the persistence registration.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ign 045)
The wire is now AimX — one tagged frame per WS text message, the same
envelope as UDS/serial/TCP; the delta really was just the envelope
(everything below the codec already rode the shared session engine).
- run_session drives the shared AimxCodec; the 507-line per-connection
WsCodec and its id↔topic maps are deleted, as are the multi-topic
Subscribe split and the protocol re-export shim.
- ClientManager delivers topic-tagged SubUpdates; the per-frame envelope
is applied by each connection's codec (payload bytes stay Arc-shared).
The pre-serialized Data frame, its server-side ts, and with_raw_payload
are gone.
- WsDispatch converges on the AimX vocabulary: record.query via a
plugged-in QueryHandler (now returning core QueryRecord rows) or the
Extensions QueryHandlerFn from with_persistence; record.list answers
{name, schema_type, entity} rows (TopicInfo now lives here).
Errors collapse to the 3-code set; auth stays at the HTTP 401.
- SnapshotProvider::snapshots(pattern) returns every cached value under
the pattern, so wildcard subscriptions late-join each covered record.
- Auto-subscribe seeds AimX sub frames with ids counting down from
u64::MAX (cannot collide with client-chosen ids; design 045 §3.6).
- ws-client builder rides AimxCodec (topic_routed_subs gone); e2e tests
rewritten to drive raw AimX frames, including golden wire frames and
ws↔AimX parity for ack/wildcard/snapshot/query/list.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…c (design 045)
The browser is now a first-class engine client: a web_sys::WebSocket-
backed Connection/Dialer pair (JS events funneled into the engine's
async recv; single-threaded wasm Send shims, same pattern and compile
guard as SendFuture) drives run_client + AimxCodec, so reply and
subscription correlation, reconnect backoff, keepalive, and the offline
queue exist exactly once — in core. The hand-rolled 835-line demux is
gone.
The #[wasm_bindgen] surface is preserved (write, query, listTopics,
onStatusChange, status, disconnect, connectBridge options; lateJoin is
retained for option-shape compatibility — snapshots are server-driven
under AimX). Subscription pumps mirror topic-tagged updates into local
records and re-subscribe when a stream ends; the queued subscribe
replays after the engine redials. WasmDb.discover speaks a one-shot
record.list request and resolves {name, schema_type, entity} rows.
Depends on aimdb-core's connector-session + remote features (the
engines cross-compile to wasm32); the aimdb-ws-protocol dep is gone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…idue (design 045) With no consumers left, the fork is gone: - aimdb-ws-protocol crate deleted (353 lines) and de-registered from the workspace members, Makefile build/test/clippy/doc targets, and the publish sequence (17 → 16 crates). - aimdb-client: dead NDJSON helpers retired with the hand-rolled client in PR #124 are deleted (RequestExt/ResponseExt/serialize_message/ parse_message/EventMessage/cli_hello — grep-confirmed zero consumers); the protocol re-export keeps RecordMetadata/WelcomeMessage/Request/ Response/Event. - design 038 §2.5/§3.9 annotated as resolved by 045; root and per-crate CHANGELOGs describe the breaking release. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nents and documentation
…ancelToken, CancelHandle, WasmWsConnection, WasmWsDialer and WsBridge
…ndshake and compatibility checks
…ns and related components
- Updated comments in `Cargo.toml` files to clarify transport features. - Revised comments in `engine.rs`, `aimx_session.rs`, and various other files to remove design references and improve clarity. - Removed outdated design references in `protocol.rs`, `query.rs`, and `codec.rs`. - Enhanced clarity in `dispatch.rs`, `builder_ext.rs`, and `http.rs` by simplifying comments. - Adjusted test descriptions in `query_handler_shape.rs` and `e2e.rs` for better understanding. - Cleaned up `ws_bridge.rs`, `registry.rs`, and `session.rs` to focus on current functionality without legacy design notes. - Updated `CHANGELOG.md` to reflect changes in `record.list` response structure and removed references to obsolete types. - Improved overall consistency and readability of comments throughout the codebase.
Reconcile main's direct-JSON-byte remote path (#192/#203) with design-048's SubUpdate.skipped loss-signal design: - Subscribe path keeps design-048's SubUpdate/skipped contract (the merged Session::subscribe trait yields BoxStream<SubUpdate>); stream.rs stays on the (Value, u64) design. - dispatch.rs combines both: SubUpdate subscribe + wildcard fan-in + WI1 hello() version gate, plus main's DispatchReply / record.get fast path. Restored the futures_util::StreamExt import main had dropped. - aimdb-bench/Cargo.toml unions both bench sets (fanout_encode + remote_json); remote_json.rs updated for Outbound::Event's new `topic` field. main's direct-JSON-bytes optimization in the live subscribe path is superseded here and left as the documented follow-up (design 048, "removing the intermediate serde_json::Value"). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The subscribe path never inspects a record update between reading it and encoding it onto the wire, yet it parsed the buffer's JSON bytes into a `serde_json::Value` (`recv_json`) only to re-serialize the tree back to bytes (`to_payload`) at the dispatch boundary — a pure parse+re-serialize round-trip per update. This is the direct-JSON-bytes follow-up design 048 flagged. `stream_record_updates` now reads owned JSON bytes via `recv_json_bytes` and yields `(Payload, u64)`; the two `stream.map` sites in the AimX dispatch build `SubUpdate` directly from those bytes. The design-048 `skipped` loss signal is untouched — it still rides each `SubUpdate`. No wire or public API change. Verified with the `remote_json` subscription-event benchmark (Tree vs Direct): the direct-bytes path sustains ~2× the throughput of the former tree path, matching the 2.06–2.49× encode figure in design 048. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lxsaah
marked this pull request as ready for review
July 24, 2026 13:28
…documentation and code
Contributor
Author
|
@thaodt I implemented the changes and the PR is ready for review again. Thanks! |
Collaborator
Contributor
Author
no issue! take your time |
thaodt
requested changes
Aug 9, 2026
…w and connection loss
Contributor
Author
|
@thaodt I implemented your suggested changes. Let's go for one additional review round, but add follow-up issues for the remaining tasks. What do you think? |
thaodt
approved these changes
Aug 12, 2026
thaodt
left a comment
thaodt
left a comment
Collaborator
There was a problem hiding this comment.
Hey I've jusst completed one final review pass.
Most of the previous concerns now look addressed, however I found two remaining code issues, see my inline comments.
I think these should be addressed before merge. I also found 2 documentation migrations that are suitable for linked follow-up issues.
Im fine with tracking those documentation sweeps as follow-up issues, provided they are linked here.
Collaborator
|
hey @lxsaah I think all good now. You can merge it |
lxsaah
added a commit
that referenced
this pull request
Sep 1, 2026
…241) * docs(persistence): teach the segment-wildcard grammar in the README `PersistenceBackend::query` moved from a prefix glob to the MQTT-style grammar subscriptions use, but the README still keyed every example with `::` and queried it with `::*`. A wildcard is a whole dot-separated segment, so `accuracy::vienna` is a single segment no pattern can partition and `accuracy::*` is a literal key that matches nothing — a reader copying the examples got zero rows and no error. Migrate the example keys to dot-separated form, then fix the patterns over them, and state the grammar once where the reader has to choose: `*` for a known depth, `#` where it varies. `query_raw` takes `#` because it feeds the AimX `record.query` handler, whose callers key at their own depth. Refs #214 * docs(persistence-sqlite): dot-separate the quickstart keys and pattern Same defect as the aimdb-persistence README, smaller surface: the quickstart persisted `accuracy::vienna` and queried it with `accuracy::*`, which matches nothing under the segment-wildcard grammar. CHANGELOG.md's `"temp::*"` stays as written — it is the historical entry describing this break, not an example to copy. Refs #214 * docs(persistence): point the query rustdoc at the canonical grammar `query_latest` taught the old grammar directly — `"accuracy::*"` as the pattern example, `"accuracy::vienna"` as its contrast. The first matches nothing now and the second is no longer a contrast, just a wildcard-free pattern. `query_range` and `query_raw` said "pattern" without defining it. State the grammar once on the trait, referring to `PersistenceBackend::query` where the crate-level statement lives, and let each method describe only what it does with the matches. `query_raw` names `QUERY_ALL_PATTERN` since that is what the AimX handler passes it. `StoredValue::record_name`'s example key loses its `::` for the same reason, and now says it is always concrete — it is the type a third-party backend author reads the contract from. Refs #214 * test(persistence): make the mock backend honor the record pattern `MockBackend::query` took `_record_pattern` and returned every row, so no test in the crate proved a pattern ever reached a backend — let alone that it was read under the segment-wildcard grammar. The six existing tests pass today and would have kept passing through a careless key rename, which is the gap review named. Filter the mock's rows through the re-exported `topic_matches`, the same matcher `PersistenceBackend::query` obliges a real backend to decide with, and dot-separate the fixture keys so a pattern can address them. `QueryParams` stays ignored: row-level filtering is `AimDbQueryExt`'s job and remains what the original six tests are about. Three cases pin the grammar at the backend-contract level, where a third-party backend author looks (aimdb-persistence-sqlite covers it against real SQLite, but only there): - `sensor.*` must not return `sensor.deep.nested` — the regression #201 was made for. Fails if the mock goes pattern-blind again. - `sensor.#` reaches that row, so the exclusion above is the pattern's doing and not a missing fixture. - a key with no dots is one segment no wildcard can address. Refs #214 * docs(session): state the real `snapshot_end` contract The client-facing doc promised the marker always arrives ("the engine reserves a sink slot for this update, so it arrives even from a burst that overran"). The reservation is real, but it only defends against overrun. The marker rides the final snapshot's frame, so a burst with no final snapshot ends without one: - **Empty burst.** `snapshots()` filters through `try_latest_as_json`, so a pattern matching nothing — or only records with no value yet — leaves `total == 0`, the loop body never runs and nothing is sent. Ordinary on a fresh database. - **Tail failed to encode.** The frame is logged and skipped, and `last` goes with it. - **Exact-topic subscribe.** Takes no snapshots at all, by design. A consumer following the old text reads "no `snapshot_end` yet" as "the burst is still arriving" and waits forever. Say instead that the marker appears when the burst produced at least one encodable snapshot, that its absence carries no information, and that `skipped` is still the loss signal. Documentation only. Making the marker unconditional needs a record-independent terminator frame — a new frame kind, `SubUpdate::data` becoming `Option`, a reservation refactor — and still could not promise delivery, since an unrecognized frame is skipped by design, which is the same property that keeps new frame kinds backward-compatible. That was considered and rejected as disproportionate. `Delivery::BurstEnd` gets the same correction internally: its "guaranteed delivery into the slot `BurstBody` kept free" holds only once a `BurstBody` has run. A single-snapshot burst rests on ordering instead — the sink predates the subscribe reply and the server sends snapshots before registering the event pump — which is what a future reader would build on. Refs #214 * docs(client): correct the unreleased `snapshot_end` changelog entry The `[Unreleased]` entry carried the same overstatement the rustdoc did — "a burst closes even when it overran the sink and even when no live event ever follows" reads as an unconditional guarantee. Unlike the historical `"temp::*"` note in aimdb-persistence-sqlite, this describes the contract about to ship, so it is corrected in place rather than left as a record. The reservation defends against overrun and that half stands; what it never covered is a burst with no final snapshot to carry the marker. aimdb-core's entry already says the flag is not an unconditional promise, and design 047 already tells consumers not to loop until it. Neither needs a change. Refs #214 * docs: tighten the prose added for the pre-release doc pass Same content, less of it: the grammar rule and the `snapshot_end` conditions were each stated once and then restated in different words. Trims ~30 lines across the READMEs, the query rustdoc, the mock's comment and the `snapshot_end` sites without dropping a fact — every case that leaves the marker unset is still named, and `*` vs `#` is still contrasted where the reader picks one. Refs #214 --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Retire
aimdb-ws-protocol— every transport speaks AimXDeletes the WebSocket-only wire protocol and converges the entire workspace on
the AimX-v3 envelope (
aimdb-core::session::aimx). One codec, one errormapping, one client demux — the WebSocket stack and the browser bridge now ride
the same session engine as UDS/serial/TCP and
aimdb-client.Implements design 047 (
docs/design/047-retire-ws-protocol-converge-on-aimx.md).Why
The workspace shipped two wire protocols for one feature. Every capability —
snapshots, acks, writes, queries, subscribe — existed twice, with two codecs and
two error vocabularies, plus a hand-rolled ~835-line browser demux duplicating
run_client. Both protocols already rode the same session engine, so the forkwas pure duplication.
Convergence removes a 353-line protocol crate, a 507-line codec, two error
mappings, and the browser demux, against a small additive AimX extension
(wildcard subscribe + two result-shape changes) that every transport now
inherits (CLI
watch 'temp.#', MCP wildcard reads, future serial/TCP browsers).What changed
Core (additive AimX extension)
#/*matches the (builder-frozen) record set once at subscribe time, merges the
matched records' update streams under one subscription id, and emits one
snapshot per matched record on open.
topic_matchmodule — MQTT-style matching over dot-separated keys, sharedby the core wildcard path and the WS fan-out bus.
SubUpdateitem type — subscription streams now carry{ topic: Option<Arc<str>>, data, skipped }so each event names the concreterecord that fired and threads buffer-lag counts through for drop detection.
topiconevent,subonsnap, an explicitsubscribedack frame (WS only; UDS/serial/TCP stay ack-implicit).record.query→{ records: [{topic, payload, ts}], total }(canonicalQueryRecordrow);RecordMetadatagainsschema_typeandentity.hello— protocol bumped2.0 → 3.0; an incompatible orversion-less peer is refused at the handshake (
RpcError::VersionMismatch).skipped; the server pump doesseq += skipped + 1; the client demux recovers loss as aseqshortfall.WebSocket connector
framing). Dispatch converged on
record.query/record.list.(topic, payload)asArc-shared updates; theper-connection codec envelopes each event (O(1) fan-out preserved).
record.queryfalls back to the persistence-registeredQueryHandlerFnwhenno custom handler is plugged in;
record.listrows are stamped with schemanames from the connector's
StreamableRegistry.WsCodec+ id↔topic maps,split_multi_topic, the pre-serializedData-frame broadcast,with_raw_payload,ClientConfig::topic_routed_subs.Browser bridge (
aimdb-wasm-adapter::ws_bridge)run_client+ClientHandleover aweb_sys::WebSocket-backedConnection/Dialer(single-threaded wasmSend/Syncwrappers). Reply andsubscription correlation now exists exactly once, in core.
write,query,listTopics,onStatusChange, offlinequeue, and reconnect are all delegated to the engine.
Deletions
aimdb-ws-protocolcrate,WsCodec,protocol.rsshims, deadaimdb-clientre-exports.
Breaking changes
2.0 → 3.0— pre-3.x clients are refused athello. Browserclients on the old ws-protocol wire must ship the rebuilt bridge; they fail
closed against a 3.0 server.
with_raw_payload,ClientConfig::topic_routed_subs.record.queryresult shape changed from{values:[{record,value,stored_at}], count}to{records:[{topic,payload,ts}],total}.Lands as one branch in a release already flagged protocol-breaking, one commit
per work item (no stacked PRs).
Testing
make checkandmake allgreen. New/updated coverage: AimX codec roundtripsuite (frames + splice byte-for-byte),
topic_matchtests,topic_leafdual-separator, version-compatibility matrix,
query_handler_shape,UDS
handshake_version, rewritten WebSockete2e.rs(subscribe ack, wildcardfan-out, late-join), and fan-out encode benchmarks.
Known limitations (accepted)
server-chosen sub ids (counted down from
u64::MAX); arun_client-basedconsumer drops events for ids it never issued, so engine clients (including the
new bridge) subscribe explicitly. No user-visible change (the pro hub already
subscribes by exact name).
time against the builder-frozen record set; MQTT-style late registration is
deferred until core grows runtime registration.