Skip to content

feat(qwp): add browser and Node.js QWP client - #62

Open
glasstiger wants to merge 228 commits into
mainfrom
ia_node_qwp
Open

feat(qwp): add browser and Node.js QWP client#62
glasstiger wants to merge 228 commits into
mainfrom
ia_node_qwp

Conversation

@glasstiger

@glasstiger glasstiger commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add a complete QWP client surface that works in both browsers and Node.js while leaving the existing ILP transports Node-only.

QWP support ships as a preview: QWP.md documents the compatibility baseline for the first QWP release, and imports from internal source paths are never supported.

Entry points

Entry point Runtime Use it for
@questdb/nodejs-client Node.js Existing Sender, including QWP ingress selected with ws::, wss::, or udp::
@questdb/nodejs-client/qwp/browser Browser Browser-safe QWP ingress, egress, authentication bootstrap, sessions, and codecs
@questdb/nodejs-client/qwp/node Node.js QWP ingress and egress with upgrade headers, TLS agents, and persistent store-and-forward
@questdb/nodejs-client/qwp Browser or Node.js Shared protocol codecs and low-level session abstractions for advanced integrations

The package root keeps the existing Node.js transports and dependencies. The browser entry point has no Node.js imports, so supporting browsers does not require breaking the existing client.

Ingress

  • browser-safe QWP codecs and WebSocket sessions
  • high-level Sender integration with fluent rows, batching, byte/interval auto-flush, commits, transactions, and ACK watermarks
  • compiled object-row table writers (sender.writer(table, schema).row({...})) for repeated rows on one schema, with the full QWP column-type set
  • fire-and-forget QWP v1 over IPv4 UDP (udp::, Node-only) behind the same fluent row API
  • automatic symbol-dictionary deltas and server batch-cap splitting
  • durable ACK negotiation, backpressure, reconnect/failover, topology-aware routing, endpoint health, and replay semantics
  • Node-only store-and-forward using the cross-client SFA persistence format, segmented replay, orphan draining, durability policies, bounded memory, and background segment maintenance. Slot locking is pure JavaScript, so a Node.js and a Java client must not use one persistence directory concurrently (see Compatibility)
  • a rejected value discards the row in progress — both its columns and its table selection — so a half-built row can never reach QuestDB

Egress

  • typed SQL binds, cancellation, query deadlines, and automatic credit replenishment
  • zstd negotiation/decompression and bounded result buffering
  • reusable column-major and row-major result views
  • reconnect/failover with server-role validation and pooled concurrent query sessions

Observability

  • immutable metrics snapshots for polling, plus onProgress, onError, and the Java-parity onSenderError rejection stream for event-driven telemetry
  • callbacks run on bounded asynchronous inboxes and never inside ACK, reconnect, or orphan-recovery protocol stacks; overflow is counted in the metrics snapshot
  • unobserved retriable rejections, terminal rejections, and abandoned data are logged by default, so background store-and-forward failures are never silent

API and platform integration

  • unified browser and Node cluster configuration with conflict validation
  • pooled QuestDB facade combining ingress and egress
  • browser authentication bootstrap for qdb_session
  • CJS/ESM subpath exports, public API contract tests, examples, migration guidance, and QWP reference documentation
  • a diagnostic benchmark suite (benchmarks/) covering encoder floors, the high-level sender, egress views, store-and-forward persistence policies, and a live end-to-end lane
  • CI: a Playwright/Chromium job that drives the built browser bundle against a local mock server, and a dormant-by-default dispatch job that queues the Enterprise JavaScript-client E2E pipeline for the exact client commit. Tests requiring a live QuestDB are owned by the server repositories, which hold the topology and authentication fixtures; the browser negotiation and session-auth contracts are covered there by QwpBrowserSessionAuthTest, QwpIngressUpgradeProcessorOnHeadersReadyTest, and QwpEgressMaxBatchRowsTest, plus the Enterprise REST/OIDC login suites

Compatibility

  • Connection strings, options, and authentication behavior of the existing http/https/tcp/tcps senders are unchanged, with the two exceptions below.
  • ILP TCP authentication is fixed on Node.js 26. auth: {keyId, token} supplies only the private scalar, and the JWK was completed with a hardcoded public point unrelated to it. Node.js accepted that inconsistent pair without validating it up to v24 and rejects it from v26 with ERR_CRYPTO_INVALID_JWK, so TCP auth failed outright on that runtime. The point is now derived from the private key. Signing only ever used the private scalar, so signatures, credentials, and auth outcomes are unchanged on every Node.js version; callers passing a complete jwk object were never affected.
  • Nullish column values changed on those senders (Client should skip columns if value is null #28). null and undefined now omit the column, which QuestDB records as NULL; most column methods previously threw a type error. On protocol v2 this also changes the wire bytes for arrayColumn(name, null), which used to emit an explicit NULL-array marker — QuestDB rejects that encoding with ARRAY_INVALID_TYPE (verified against 9.4.3), so omitting it is itself a fix. A row whose every value is nullish now fails when the row is closed rather than at the column call. Code that relied on the throw as a data-quality guard should validate before calling the sender.
  • Browser mode does not provide disk persistence; store-and-forward is Node-only.
  • A Node.js client and a Java client must not use one store-and-forward directory at the same time. The Java client guards slots with flock/LockFileEx; the Node.js client uses a pure-JavaScript directory lock and cannot participate in those kernel locks, so neither sees the other. The persistence format stays cross-client for sequential handoff — a directory written by one runtime can be opened by the other once the first has closed it — and two Node.js processes still exclude each other. Depending on a native addon for kernel locks was the alternative, and it left store-and-forward broken on any platform or Node.js major without a prebuilt binary.
  • Browser-only capabilities are negotiated in protocol messages because browser WebSocket APIs cannot set custom upgrade headers.

Dependencies

  • new runtime dependency: ws (Node WebSocket transport). There is no native dependency; store-and-forward locking is pure JavaScript
  • fzstd is bundled into the build output for egress decompression; THIRD_PARTY_NOTICES.md records its license

Resolved issues

Validation

  • full test suite, including containerized integration coverage: 27 files / 612 tests passed
  • targeted reconnect and SFA interoperability suite: 109 tests passed
  • benchmark self-checks (pnpm vitest run benchmarks): 3 files / 14 tests passed
  • pnpm typecheck
  • pnpm typecheck:qwp-browser
  • pnpm typecheck:bench
  • pnpm eslint
  • pnpm lint:bench
  • pnpm build
  • built CJS and ESM runtime smoke tests

Dependencies and provenance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

State-machine builder Client should skip columns if value is null

1 participant