RDBC-1104 Enhance bulk insert perfromance - #306
Merged
poissoncorp merged 6 commits intoAug 20, 2026
Conversation
Utils.entity_to_dict decides what a document looks like on the wire, for both the session and the bulk insert path. Before changing how it is implemented, capture what it currently produces: nested objects, to_json, datetime and timedelta formats, the three flavours of enum, tuple and set handling, dict key coercion, which types survive as themselves, non-finite floats, non-ASCII text, shared references, circular reference detection and the unsupported-value errors. Every case is also compared against the reference round trip through a JSON string, so the tests double as a differential oracle for that change.
Utils.entity_to_dict turned an entity into a plain dict by serializing it to a
JSON string and parsing it back, and the callers then serialized that dict again.
Every stored document was therefore encoded twice and parsed once.
Walk the object graph instead. The conversion has to keep what the round trip
gave for free, so it follows the same order json checks types in, writes scalar
subclasses by their built-in value, coerces dict keys the way json does, and
tracks containers by identity to report a circular reference rather than
recursing until the stack ends.
documents per second, 10,000 documents, one thread, median of three
1024 dimensions 991 -> 1,929
384 dimensions 2,613 -> 5,363
peak allocation per store, 1024 dimensions 110,673 B -> 85,986 B
The bytes handed to the server are unchanged.
The flush condition was "the buffer is over the threshold, or the previous
handover has already finished". Handing a buffer over was a put on an unbounded
queue, which completes immediately, so the second half was almost always true.
The 1 MiB threshold therefore rarely decided anything: chunk size came out of
thread timing instead, and it moved whenever unrelated code got faster, from
352 KiB before this branch to 684 KiB once conversion stopped being the
bottleneck.
Drop the second half so the chunk size is the one that was asked for.
average chunk 684 KiB -> 1,051 KiB
peak allocation per store, 1024 dimensions 85,986 B -> 80,098 B
documents per second, 1024 dimensions 1,929 -> 1,914 (within noise)
The gain is not throughput: it is that a store which does not flush no longer
copies the buffer, and that chunk size stops depending on how fast the rest of
the client happens to be.
A finished buffer took two copies to reach the thread sending the request. The
flush deepcopied it and cleared the original, then the queue copied it again into
an immutable bytes. Both copies are of a buffer just over 1 MiB.
Both had a reason, and it was the same reason: the flush aliased the buffer and
then cleared it in place, so whatever had already been handed over was wiped
(RDBC-644), and the copies were what kept the sender's data alive. Stop clearing
and reusing one buffer: hand this one over and start a fresh one. The caller then
holds the only reference to the new buffer, the sender holds the only reference to
the old one, and neither copy has anything left to protect.
That also removes the handover from the thread pool, which existed to keep the
copying off the calling thread and now has nothing to do: a put on the queue is
something the caller can do itself.
documents per second, 10,000 documents, one thread, median of three
1024 dimensions 1,914 -> 2,009
384 dimensions 5,157 -> 5,603
Both come out about 0.2 s shorter over the same 200-odd chunks, which is the
per-chunk work that went away.
The tests hold the ownership rule that replaces the copies. One watches every
buffer that leaves, snapshots it on arrival and checks at the end that nothing
wrote into it afterwards, that no two handovers are the same object, and that the
reassembled stream is the whole payload. The other loads 5 MiB through requests
and a real server, so several buffers are flushed mid-stream, and checks that the
documents arrive intact; buffers used to be turned into immutable bytes right
before they reached requests (RDBC-706) and are now handed over as they are.
…ound
The queue holding finished buffers had no maximum size, so a caller writing
faster than the server reads accumulated the difference in memory. Measured
against a stream that stopped being read: 23,030 bytes per document stayed on
the client, which is the whole payload, 46.8 GiB for a 2.2M document load. The
.NET and Java clients write into the request stream and block on the socket.
Bound the queue to MAX_BUFFERS_TO_FLUSH buffers, so at most that many finished
buffers of just over 1 MiB can wait, and hand each one over with a put that waits
for a free slot. While waiting, keep checking that the thread sending the request
is still alive: if it failed, report the abort, and if it finished, stop waiting
for a reader that is not coming back.
This costs nothing when the server keeps up, which is the normal case. Against a
real server, 10,000 documents of 1024 dimensions, alternating runs:
bounded to 8 median 1,864 documents per second
unbounded median 1,885 documents per second
peak queue depth 2 to 5 buffers in both
The bound is never reached there, so the put never actually waits. What it buys is
that the memory cannot grow without one.
While in this class: the exception the queue raises lives in queue, not in the
_queue extension module, which only provides Empty and SimpleQueue, so both are
now imported from the same place. The semaphore next to the queue was assigned
and never read.
BulkInsertOptions has taken a use_compression flag all along, and nothing ever
read it. The value landed in a private field that had no readers, while the
command copied a public attribute that __init__ pinned to False, so the option
could not be turned on through the public API at all, and there was no gzip in
the request either way.
Read the option, and compress the outgoing buffers as one gzip stream, flushing
after each buffer so the server keeps processing documents while the rest is
still being written. This is the shape the .NET and JVM clients use: one
compressed stream over the whole request body, with the encoding declared in the
header.
The level is the one .NET uses, Fastest, because the client is the slow side of
this insert and zlib's default level makes it slower still. 5,000 documents of
1024 dimensions with distinct vectors, against a real server:
zlib default level 668 documents per second, 9,489 B per document
Fastest 1,398 documents per second, 10,328 B per document
9% more bytes for twice the throughput, on a load that was running at 2,000
documents per second before compression entered the picture. Uncompressed the
same document is 22,795 B, so a 2.2M document load goes from 46.7 GiB to
21.2 GiB.
Verified against 7.2: the documents arrive intact and their contents match.
Compression stays off unless asked for, so the default stays byte for byte what
it was.
poissoncorp
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
https://issues.hibernatingrhinos.com/issue/RDBC-1104
What changed
Bulk insert was bound by the client: each document was serialized to JSON twice and parsed once, each finished buffer was copied twice on its way to the request, and the queue holding those buffers had no limit. It now serializes once, moves buffers instead of copying them, blocks when the queue is full, and
use_compressionactually compresses. Throughput is 2.04x at 1024-dimension documents and 2.16x at 384, with the same bytes on the wire unless compression is asked for.One commit per change, each with its tests:
813ba79Pin the conversion semantics - whatentity_to_dictproduces, case by case, checked against the old round trip.afd28cbConvert without a JSON round trip - encode once instead of twice: 991 -> 1,929 docs/s at 1024 dims.session.store()goes through the same conversion.49c0f41Flush on buffer size alone - the second half of the condition was always true here: chunks 684 KiB -> 1,051 KiB.b1b4e52Hand the buffer over instead of copying it twice - both copies existed to protect a buffer that was aliased and cleared: 1,914 -> 2,009 docs/s.d7a2dc6Bound the outbound queue - a stalled stream used to keep the whole payload client side, 23,030 B per document; now 8 buffers at most and the caller waits.ebc716fCompress when asked -use_compressionwas assigned in three places and read in none; one gzip stream at .NET'sFastestlevel: 22,795 -> 10,328 B per document.Checklist