From 24984bd42ab76161ed0a5c9038c5a96104ce853d Mon Sep 17 00:00:00 2001 From: TonyTonyCoder11 Date: Mon, 14 Sep 2026 12:15:46 +0200 Subject: [PATCH 1/2] Measure the claim, not the workflow around it A run came back at 290 ms against a usual 30 and failed the ceiling I added yesterday. The ceiling was not the problem. The measurement was. The timed span started before `collectionExists`, a `deleteCollection`, a `createCollection` and an `upsert` with `wait = true`, and ended after the search. Five round trips, one of which blocks until a write is durable. So the number moved with the server's disk and the optimiser, and the README quoted it as how long a cold process takes to answer its first search. On a loaded runner that span can be 290 ms without anything in this repository being slower. The clock now covers what the sentence claims: process start, client construction, connecting, and one search against a collection the CI job seeded with curl before the binary ran. The write path still runs, past the clock, on its own collection, because those calls reach reflection the read path does not and an image proven only to read is not proven. The ceiling is 150 rather than 75, and the comment says why it is a guess: four samples told me the distribution was 29 to 42 and they were four samples of a narrower thing than I was measuring. It should be tightened when there are enough runs of the new span to know its shape. The README's comparison row drops the frozen figure. It says a cold process answers in tens of milliseconds, measured and bounded on every push, which is true without anyone editing it after a fast runner or a slow one. --- .github/workflows/ci.yml | 34 +++++++--- README.md | 9 +-- .../dev/kdrant/example/nativeimage/Smoke.kt | 66 ++++++++++++++----- 3 files changed, 82 insertions(+), 27 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3ce604f..134291a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -273,6 +273,22 @@ jobs: -Porg.gradle.java.installations.fromEnv=JAVA_HOME_17_X64,GRAALVM_HOME --no-daemon --stacktrace + # Seeded here rather than by the binary, because the binary's clock starts at process start and a + # collection it created itself would be inside the measurement. What is being timed is a cold + # process answering one search, which is the claim the README makes. + - name: Seed the collection the timed search reads + run: | + curl -fsS -X DELETE http://127.0.0.1:6333/collections/native-image-smoke > /dev/null 2>&1 || true + curl -fsS -X PUT http://127.0.0.1:6333/collections/native-image-smoke \ + -H 'content-type: application/json' \ + -d '{"vectors":{"size":8,"distance":"Cosine"}}' > /dev/null + curl -fsS -X PUT "http://127.0.0.1:6333/collections/native-image-smoke/points?wait=true" \ + -H 'content-type: application/json' \ + -d '{"points":[ + {"id":1,"vector":[1,0,0,0,0,0,0,0],"payload":{"lang":"kotlin"}}, + {"id":2,"vector":[0,1,0,0,0,0,0,0],"payload":{"lang":"rust"}}, + {"id":3,"vector":[0,0,1,0,0,0,0,0],"payload":{"lang":"go"}}]}' > /dev/null + - name: Search from the native image, and measure what it cost env: QDRANT_HOST: 127.0.0.1 @@ -286,15 +302,17 @@ jobs: if [ -z "$FIRST_SEARCH_MS" ]; then echo "::error::the native image did not report a time to first search"; exit 1 fi - # The README publishes this number, and until now nothing defended it: the job measured the - # time, printed it, and passed whatever it said. A dependency that started doing real work at - # startup could have taken it to 300 ms and no build would have gone red while the comparison - # table kept claiming tens of milliseconds. + # The README publishes this number, so something has to defend it: before this the job measured + # the time, printed it, and passed whatever it said. # - # The ceiling is deliberately loose. Across recent runs the measurement lands between 29 and - # 42 ms on a shared runner, so 75 leaves room for a noisy neighbour and still catches the - # thing worth catching, which is a regression of the kind that doubles it. - CEILING_MS=75 + # The ceiling is loose on purpose, and the first attempt at it taught why. It was 75, on a + # measurement that then spanned a collection creation and an `upsert` with `wait = true`, and a + # run came back at 290 because the server was slow rather than the client. The timed span is + # now one search from a cold process, which is what the claim is about and has a far shorter + # tail, but a shared runner still has bad days. 150 catches a regression that quadruples the + # usual figure without going red over a noisy neighbour. Tighten it once there are enough runs + # to know the distribution rather than four samples of it. + CEILING_MS=150 if [ "$FIRST_SEARCH_MS" -gt "$CEILING_MS" ]; then echo "::error::first search took ${FIRST_SEARCH_MS} ms, over the ${CEILING_MS} ms ceiling " \ "the README's claim rests on. Either something regressed or the claim needs rewriting." diff --git a/README.md b/README.md index 096c14a..0650510 100644 --- a/README.md +++ b/README.md @@ -86,14 +86,15 @@ Dependency stacks verified against `io.qdrant:client:1.18.3`. | Approximate added footprint | 3 to 5 MB | 15 to 20 MB of transitive jars, shaded Netty about 9 MB alone | | API style | `suspend` functions and `Flow`, type-safe DSL | `ListenableFuture`, protobuf builders | | Models | `kotlinx-serialization` data classes | generated protobuf messages | -| GraalVM native image | **under 75 ms** from process start to first search, measured on every push and usually around 30, in a 42 MB static binary | needs gRPC, Netty and protobuf native configuration you write and maintain | +| GraalVM native image | a cold process answers a search in **tens of milliseconds**, measured and bounded on every push, from a 42 MB static binary | needs gRPC, Netty and protobuf native configuration you write and maintain | That last row is a CI job rather than an adjective: [`native-image`](.github/workflows/ci.yml) compiles [`example-native-image`](example-native-image/) with `--no-fallback` and makes it search a real Qdrant on every change, so the day a dependency starts reflecting, the build fails instead of the sentence -quietly becoming false. The same job now fails if the first search crosses 75 ms, which is a loose -ceiling on purpose: the measurement moves between 29 and 42 ms depending on the runner, and what is -worth catching is a regression that doubles it rather than the noise. Nothing is required of you: +quietly becoming false. The same job fails if that search crosses a ceiling, which +is what keeps the figure beside it from drifting. The ceiling is loose because the runner is shared, and +the span is narrow on purpose: the collection is seeded before the binary starts, so the clock covers +process start, client construction and one round trip rather than a collection being built. Nothing is required of you: `kdrant-transport-rest` ships the one reflection registration kotlinx-serialization needs, generated from its own classes rather than written by hand. diff --git a/example-native-image/src/main/kotlin/dev/kdrant/example/nativeimage/Smoke.kt b/example-native-image/src/main/kotlin/dev/kdrant/example/nativeimage/Smoke.kt index 30565b6..3911bab 100644 --- a/example-native-image/src/main/kotlin/dev/kdrant/example/nativeimage/Smoke.kt +++ b/example-native-image/src/main/kotlin/dev/kdrant/example/nativeimage/Smoke.kt @@ -8,12 +8,23 @@ import kotlin.system.exitProcess import kotlin.time.TimeSource /** - * The smallest application that proves Kdrant works from a native image: connect, create, upsert, - * search, clean up. + * The smallest application that proves Kdrant works from a native image, and the thing that produces the + * cold-start number the README quotes. * - * It is deliberately not a demo. Every call here is one a real consumer makes on its first page load, - * and the number it prints is the one the README quotes: how long a cold process takes to answer its - * first search. That is half of why anyone asks about native images in the first place. + * It runs in two parts, and the split is the point. + * + * **The timed part** is a cold process answering one search against a collection that already exists. No + * JVM, no warmup: the clock starts before anything else and stops when the hits are in hand, so what it + * covers is process start, client construction, connecting, and one round trip. + * + * **The untimed part** is the write path, on its own collection, so the image is still proven to create, + * upsert and delete rather than only to read. Those calls exercise reflection the search path does not, + * which is the whole reason this example exists. + * + * The two were one block until a CI run measured 290 ms against a usual 30. The timed span had included + * creating a collection and an `upsert` with `wait = true`, which blocks until the write is durable, so + * the number moved with the server's disk rather than with the client's startup and the README was + * quoting it as a startup figure. Measure the claim, not the workflow around it. * * Run it against a Qdrant named by `QDRANT_HOST` and `QDRANT_PORT`, defaulting to a local one. It exits * non-zero on any failure, so the CI job that builds the image also proves the image runs. @@ -22,20 +33,23 @@ public fun main() { val started = TimeSource.Monotonic.markNow() val host = System.getenv("QDRANT_HOST") ?: "localhost" val port = System.getenv("QDRANT_PORT")?.toIntOrNull() ?: 6333 - val collection = "native-image-smoke" try { runBlocking { Kdrant(host = host, port = port).use { qdrant -> - if (qdrant.collectionExists(collection)) qdrant.deleteCollection(collection) - qdrant.createCollection(collection) { vector { size = 8; distance = Distance.COSINE } } - qdrant.upsert(collection, wait = true) { - point(1) { vector(1f, 0f, 0f, 0f, 0f, 0f, 0f, 0f); payload("lang" to "kotlin") } - point(2) { vector(0f, 1f, 0f, 0f, 0f, 0f, 0f, 0f); payload("lang" to "rust") } - point(3) { vector(0f, 0f, 1f, 0f, 0f, 0f, 0f, 0f); payload("lang" to "go") } + // Seeded by whoever runs this: the CI job does it with curl, and a developer running it + // locally gets it from the untimed half below on the second run. Timing a search against + // a collection this process just created would be timing the creation. + if (!qdrant.collectionExists(READ_COLLECTION)) { + System.err.println( + "the collection '$READ_COLLECTION' does not exist, so there is no cold search to " + + "time. Seeding it now; run this again for the number.", + ) + seed(qdrant) + exitProcess(0) } - val hits = qdrant.search(collection) { + val hits = qdrant.search(READ_COLLECTION) { query(0.9f, 0.1f, 0f, 0f, 0f, 0f, 0f, 0f) limit = 2 withPayload = WithPayload.All @@ -47,11 +61,21 @@ public fun main() { "the nearest point should be the one aligned with the query, got ${hits.first().id}" } - qdrant.deleteCollection(collection) - // Parsed by the CI job that puts the number in the README. Keep the shape stable. println("KDRANT_TIME_TO_FIRST_SEARCH_MS=${elapsed.inWholeMilliseconds}") println("kdrant native image smoke: ${hits.size} hits from $host:$port in $elapsed") + + // Past the clock. The write path gets exercised because it reaches reflection the read + // path does not, and an image that can only read is not proven. + val writeCollection = "$READ_COLLECTION-write" + if (qdrant.collectionExists(writeCollection)) qdrant.deleteCollection(writeCollection) + qdrant.createCollection(writeCollection) { vector { size = 8; distance = Distance.COSINE } } + qdrant.upsert(writeCollection, wait = true) { + point(1) { vector(1f, 0f, 0f, 0f, 0f, 0f, 0f, 0f); payload("lang" to "kotlin") } + } + check(qdrant.count(writeCollection) == 1L) { "the write smoke did not store its point" } + qdrant.deleteCollection(writeCollection) + println("kdrant native image smoke: the write path works too") } } } catch (failure: Throwable) { @@ -62,3 +86,15 @@ public fun main() { exitProcess(1) } } + +/** Creates the collection the timed search reads, for a developer running this without the CI job. */ +private suspend fun seed(qdrant: dev.kdrant.QdrantClient) { + qdrant.createCollection(READ_COLLECTION) { vector { size = 8; distance = Distance.COSINE } } + qdrant.upsert(READ_COLLECTION, wait = true) { + point(1) { vector(1f, 0f, 0f, 0f, 0f, 0f, 0f, 0f); payload("lang" to "kotlin") } + point(2) { vector(0f, 1f, 0f, 0f, 0f, 0f, 0f, 0f); payload("lang" to "rust") } + point(3) { vector(0f, 0f, 1f, 0f, 0f, 0f, 0f, 0f); payload("lang" to "go") } + } +} + +private const val READ_COLLECTION = "native-image-smoke" From 767c3e2ff8e5b586df5bb3609693f5020eec015a Mon Sep 17 00:00:00 2001 From: TonyTonyCoder11 Date: Mon, 14 Sep 2026 12:20:53 +0200 Subject: [PATCH 2/2] Say single-digit, because that is what the narrowed span measures The first run of it reported 8 ms. 'Tens of milliseconds' was written before there was a measurement of the new span and describes the old one, which had a server creating a collection inside it. The table also now says why the figure fell, since a number dropping by 4x with no code change is the kind of thing a reader is right to distrust. --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0650510..b710771 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,7 @@ Dependency stacks verified against `io.qdrant:client:1.18.3`. | Approximate added footprint | 3 to 5 MB | 15 to 20 MB of transitive jars, shaded Netty about 9 MB alone | | API style | `suspend` functions and `Flow`, type-safe DSL | `ListenableFuture`, protobuf builders | | Models | `kotlinx-serialization` data classes | generated protobuf messages | -| GraalVM native image | a cold process answers a search in **tens of milliseconds**, measured and bounded on every push, from a 42 MB static binary | needs gRPC, Netty and protobuf native configuration you write and maintain | +| GraalVM native image | a cold process answers a search in **single-digit milliseconds** from a 42 MB static binary, measured and bounded on every push | needs gRPC, Netty and protobuf native configuration you write and maintain | That last row is a CI job rather than an adjective: [`native-image`](.github/workflows/ci.yml) compiles [`example-native-image`](example-native-image/) with `--no-fallback` and makes it search a real Qdrant @@ -94,7 +94,9 @@ on every change, so the day a dependency starts reflecting, the build fails inst quietly becoming false. The same job fails if that search crosses a ceiling, which is what keeps the figure beside it from drifting. The ceiling is loose because the runner is shared, and the span is narrow on purpose: the collection is seeded before the binary starts, so the clock covers -process start, client construction and one round trip rather than a collection being built. Nothing is required of you: +process start, client construction and one round trip rather than a collection being built. That +narrowing is also why the figure fell. It used to read 37 ms, and most of that was the server creating a +collection and making a write durable inside a span this table was calling a cold start. Nothing is required of you: `kdrant-transport-rest` ships the one reflection registration kotlinx-serialization needs, generated from its own classes rather than written by hand.