feat(qwp): add browser and Node.js QWP client - #62
Open
glasstiger wants to merge 228 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a complete QWP client surface that works in both browsers and Node.js while leaving the existing ILP transports Node-only.
QWP support ships as a preview:
QWP.mddocuments the compatibility baseline for the first QWP release, and imports from internal source paths are never supported.Entry points
@questdb/nodejs-clientSender, including QWP ingress selected withws::,wss::, orudp::@questdb/nodejs-client/qwp/browser@questdb/nodejs-client/qwp/node@questdb/nodejs-client/qwpThe package root keeps the existing Node.js transports and dependencies. The browser entry point has no Node.js imports, so supporting browsers does not require breaking the existing client.
Ingress
Senderintegration with fluent rows, batching, byte/interval auto-flush, commits, transactions, and ACK watermarkssender.writer(table, schema).row({...})) for repeated rows on one schema, with the full QWP column-type setudp::, Node-only) behind the same fluent row APIEgress
Observability
onProgress,onError, and the Java-parityonSenderErrorrejection stream for event-driven telemetryAPI and platform integration
qdb_sessionbenchmarks/) covering encoder floors, the high-level sender, egress views, store-and-forward persistence policies, and a live end-to-end laneQwpBrowserSessionAuthTest,QwpIngressUpgradeProcessorOnHeadersReadyTest, andQwpEgressMaxBatchRowsTest, plus the Enterprise REST/OIDC login suitesCompatibility
http/https/tcp/tcpssenders are unchanged, with the two exceptions below.auth: {keyId, token}supplies only the private scalar, and the JWK was completed with a hardcoded public point unrelated to it. Node.js accepted that inconsistent pair without validating it up to v24 and rejects it from v26 withERR_CRYPTO_INVALID_JWK, so TCP auth failed outright on that runtime. The point is now derived from the private key. Signing only ever used the private scalar, so signatures, credentials, and auth outcomes are unchanged on every Node.js version; callers passing a completejwkobject were never affected.nullandundefinednow omit the column, which QuestDB records as NULL; most column methods previously threw a type error. On protocol v2 this also changes the wire bytes forarrayColumn(name, null), which used to emit an explicit NULL-array marker — QuestDB rejects that encoding withARRAY_INVALID_TYPE(verified against 9.4.3), so omitting it is itself a fix. A row whose every value is nullish now fails when the row is closed rather than at the column call. Code that relied on the throw as a data-quality guard should validate before calling the sender.flock/LockFileEx; the Node.js client uses a pure-JavaScript directory lock and cannot participate in those kernel locks, so neither sees the other. The persistence format stays cross-client for sequential handoff — a directory written by one runtime can be opened by the other once the first has closed it — and two Node.js processes still exclude each other. Depending on a native addon for kernel locks was the alternative, and it left store-and-forward broken on any platform or Node.js major without a prebuilt binary.Dependencies
ws(Node WebSocket transport). There is no native dependency; store-and-forward locking is pure JavaScriptfzstdis bundled into the build output for egress decompression;THIRD_PARTY_NOTICES.mdrecords its licenseResolved issues
Null or undefined column and symbol values are omitted across the existing ILP senders and the new QWP senders.
Fixes Client should skip columns if value is null #28
The new QWP sender introduces the sender.write().row() API.
Fixes State-machine builder #60
Validation
pnpm vitest run benchmarks): 3 files / 14 tests passedpnpm typecheckpnpm typecheck:qwp-browserpnpm typecheck:benchpnpm eslintpnpm lint:benchpnpm buildDependencies and provenance