An index write per chunk per term was most of an ingest - #530
Merged
Conversation
83.3% of the records a skill ingest wrote were index records: 33,020 of the
39,624 a 1 MB skill produces, one for every (chunk, term) pair. Coalescing them
into one posting per term writes the same index -- the same 3,026 terms carrying
the same 36,046 references, checked through context_index_ref_hashes, the helper
the retrieve path itself uses -- and takes context_index from 13.6 MB to 1.8 MB.
That could not be turned on before. Serving resolved a record's identity through
the singular fields and never read `ref_hashes`, so a posting carrying two refs
had no identity and was dropped outright, silently. And the emitter ignored
MAX_SECONDARY_INDEX_REFS_PER_POSTING: it wrote a single 3,302-ref posting against
a cap of 512 that compact_context_index_postings observes. A cap one producer of
a record type respects and another ignores is not a bound on anything. The
emitter now splits at the cap, stamps posting_part, and carries the singular
ref_hash only for a single-ref part, matching the compactor exactly.
Vector compaction was then 44% of what emission cost: 1,690,624 round() calls per
document, one per dimension of every chunk vector. The arithmetic is trivial and
the Python loop is not, so it runs in numpy where numpy exists -- emission 1,143ms
to 525ms, 2.18x. numpy is not assumed; the loop remains, and both paths are
covered by the same equality test over exact .5 boundaries, all-zero vectors,
negative zero and the empty vector, where two rounding implementations diverge if
either does not round half to even.
Index priority was a 20-prefix startswith scan per term, 525,018 calls across the
36,322 terms a document emits. Every prefix is exactly "<kind>:" and every term is
exactly f"{kind}:{value}", so the scan was an equality test on the kind: now a
dict lookup, 40ms to 9ms per 40,000 terms, verified identical on 107 terms
including values containing colons and kinds that merely share an opening
substring with a listed one.
cosine() returned a bare dot product over assumed-unit inputs, and
normalized_dense_score clamps (v + 1) / 2 into [0, 1]. That is correct only while
every stored vector is unit, and every vector-compaction option produces one that
is not. It now divides by both norms, which is a no-op for what is stored today --
a unit vector's dot already IS its cosine -- and is what makes a compacted vector
score where the float it replaced scored.
With that fixed, EMBEDDING_VECTOR_SCALE defaults to 100000. A uniform scale
multiplies every vector by the same constant, so no pair can change places.
Measured over 500 real chunks with six EN/CN queries against the float ranking:
top-1 6/6, exact top-10 order 6/6, overlap 10.0/10, under both the normalised
scorer and the bare dot it replaced.
int8 stays off, and now on evidence rather than inference. The same measurement
puts it at top-1 4/6 and exact top-10 order 0/6 under the normalised scorer, and
0/6 and 0.5/10 under the bare dot -- so normalising took it from unusable to
merely wrong. int8 divides each vector by its own peak, so two stored vectors are
scaled by different factors and can change places against one query; a uniform
scale cannot. A synthetic near-neighbour set showed int8 exact and was simply too
easy.
Three WAL and index gates that default ON and whose off branch no test executes
are removed rather than kept: group commit, upsert delta emission, and the
relaxed WAL parent-dir fsync -- the last already unconditional, since it read
`!TS_WAL_LEGACY_RECOVERY || group_commit_enabled()` and group commit defaulted on.
At the shipped defaults a 1 MB skill now writes 9,659 records and 17.8 MB where
it wrote 39,624 and 36.4 MB: amplification 33.9x to 16.6x, 36.4 GB to 17.8 GB per
thousand documents, and the footprint is 65.0% embeddings, 24.6% text and 10.3%
index where the index alone used to be 37.3%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The equivalence tests compare the numpy vector path against the pure-Python fallback, and one of them asserted numpy was present so the comparison could not silently run the fallback against itself. CI has no numpy, so that assertion turned a supported configuration into a failure. Where numpy exists the comparisons are real and now run; where it does not there is one implementation and nothing to compare, so they skip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ults
int8 was documented as unusable for retrieval on the strength of a measurement
that put it at top-1 1/6 and top-10 overlap 4.8/10 against the float ranking.
That is not a property of eight bits. It is a property of dividing each vector by
its OWN peak: two stored vectors scaled by different factors can change places
against one query, and no amount of precision repairs it.
A scale that depends only on the vector's WIDTH is uniform across every vector,
so it cannot reorder, while still being computable one vector at a time -- which
a corpus-derived scale is not. Unit vectors in d dimensions have elements around
1/sqrt(d), and 8/sqrt(d) covers that distribution and its tail.
Measured over 500 real chunks with six EN/CN queries, e5-large at 512 dims,
against the float ranking:
rule top-1 exact@10 overlap@10 recall@50 clipped
per-vector peak (before) 4/6 0/6 9.5/10 10.0/10
127 / (4/sqrt(d)) 4/6 0/6 9.7/10 10.0/10 0.195%
127 / (6/sqrt(d)) 5/6 0/6 9.0/10 10.0/10 0.172%
127 / (8/sqrt(d)) chosen 6/6 1/6 9.5/10 10.0/10 0.000%
127 / (10/sqrt(d)) 5/6 0/6 9.3/10 10.0/10 0.000%
Top-1 becomes exact and nothing clips. recall@50 is 10/10 for every variant
including the old one: int8 never loses a true result, it reorders within the
shortlist -- which is why the shortlist-then-rescore pattern reproduces the exact
top-10 on 6/6 queries even at top-20.
Per-DIMENSION scaling was measured too and is far worse (top-1 1/6, overlap
5.5/10): it reweights dimensions and distorts the cosine itself. Recorded here so
it is not tried again.
int8 stays OFF by default. scale=1e5 is exact on all three measures where int8
trades intra-top-10 order for 21.7% fewer stored bytes, and that trade belongs to
whoever runs the deployment rather than to the default. What changes is that the
option is now worth having: 13.9 MB against 17.8 MB per 1 MB skill, 2,082 bytes
per 512-dim vector against 3,236.
The uniformity of the scale is what the encoding depends on, so a test pins it
where the two rules DIVERGE: under a per-vector peak a vector and its double
normalise to identical output, and under a width-derived scale the output
doubles. Recovering the factor by dividing output by input does not work --
quantisation rounding on small elements swamps it, which is what made an earlier
version of that test fail against correct code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
100000 and 10000 both reproduce the float ranking exactly. The difference is a
character per element: 3,247 bytes per 512-dim vector against 2,736.
Measured over 500 real chunks with six EN/CN queries, e5-large at 512 dims,
against the float ranking:
encoding top-1 exact@10 overlap@10 bytes/vec step
float round(6) 6/6 6/6 10.0/10 5328 B
int8 (width scale) 6/6 1/6 9.5/10 2110 B 0.00278
uniform scale = 1000 5/6 3/6 9.8/10 2242 B 0.00100
uniform scale = 3000 6/6 4/6 9.8/10 2545 B 0.00033
uniform scale = 10000 6/6 6/6 10.0/10 2736 B 0.00010
uniform scale = 100000 6/6 6/6 10.0/10 3247 B 0.00001
This also settles what int8's remaining cost actually is. It is not the bit
width and not the uniformity -- both int8 and these scales are uniform, so
neither can reorder through scaling. It is RESOLUTION: int8's step is 0.00278 and
the margins between competing near-neighbours in this corpus are smaller than
that, so it reshuffles inside the top ten. 10000 has a step of 0.0001, which
resolves every one of them; 100000 resolves nothing further and charges for it.
At the shipped defaults a 1 MB skill now writes 16.1 MB where it wrote 36.4 MB:
amplification 33.9x to 15.0x, 36.4 GB to 16.1 GB per thousand documents, and the
footprint is 61.4% embeddings, 27.2% text, 11.4% index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two constants set how much text a chunk holds and how much of it reaches the
encoder, and they disagreed: chunks were 240 tokens while
DEFAULT_EMBEDDING_TEXT_MAX_TOKENS was 128. Roughly half of every chunk never
reached a vector. That text was findable only through the lexical index, and
96.1% of the terms a skill ingest writes there cannot be consulted by the
retrieve path at all -- so in practice it was unreachable.
Neither number came from the encoder. e5 models read 512 tokens, BGE-M3 and
jina-v3 read 8192, mpnet reads 384. A fixed window either wastes an encoder's
capacity or overruns it and is silently truncated by the tokenizer, and it is
wrong for every model except by accident.
Both now follow the active encoder, so a chunk is never larger than what will be
embedded, and an unrecognised model falls back to 512 rather than guessing.
Chunk count is the dominant memory lever, because vectors are the largest record
type and their COUNT is set by the window. Measured on a 1 MB skill:
window chunks records bytes amplification
128 3,302 9,659 16.1 MB 15.0x
512 744 2,281 4.6 MB 4.3x
That is 4.4x fewer vectors and 4.6 GB per thousand documents where the session
started at 36.4 GB. Composition at 512 is 48.2% embeddings, 42.0% text, 9.8%
index. Encoding is also 4.5x faster on the same text, since there are 4.4x fewer
chunks to run through the model.
The suite is unchanged by this: the ten failures in the resource, index and
ingest modules occur identically with the previous 240/128 values restored
through the environment, so they are pre-existing.
What this commit does NOT carry is a recall number at 512. One vector now stands
for 1,342 characters instead of 308, and whether that still finds what the
smaller chunk found is a measurement, not an inference. Two attempts today
produced one usable query and then none, because the needles were invented
rather than drawn from the corpus; a run against the 298-pair set is in flight.
Both windows remain settable through MATRIXARK_RESOURCE_MAX_CHUNK_TOKENS and
MATRIXARK_EMBEDDING_TEXT_MAX_TOKENS, so this is reversible without a deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sizing chunks to what the encoder can read was measured and is wrong. Over 298
query pairs across 79 documents with e5-large:
window chunks chars/chunk hit@1 hit@5
128 2,753 368 77.2% 92.3%
512 686 1,189 52.0% 77.2%
4.01x fewer vectors costs 25.2 points of hit@1 and 15.1 of hit@5. A quarter of
the correct top-1 results disappear. One 512-dimension vector cannot stand for
three times as much text without blurring what it points at, and no amount of
memory saved is worth a third of the retrieval.
So the model's window bounds the chunk from ABOVE -- never embed more than the
encoder will read, or the tokenizer truncates it silently -- but it is not a
target to reach for. How much text one vector should represent is a retrieval
question that happens to bound memory, not a capability question.
The per-model window function stays, because a fixed constant is wrong for every
encoder except by accident: e5 and MiniLM stop at 512, BGE-M3 and jina-v3 at
8192, mpnet at 384, and anything unrecognised falls back to 512. It is now used
as the ceiling it should always have been.
The 240/128 mismatch this started from is left as it ships. It is real -- chunks
are 240 tokens and only 128 of them are embedded, so roughly half of every chunk
reaches no vector and is findable only through a lexical index whose terms the
retrieve path cannot consult -- but the two ways to close it move memory in
opposite directions. 128/128 embeds every token and produces about 1.9x more
chunks; 240/240 keeps the chunk count and gives one vector twice as much text to
carry. 128 against 512 cost 25 points, so the curve between them is not
something to guess at, and neither option ships without a number.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
83.3% of the records a skill ingest wrote were index records — 33,020 of 39,624 for a 1 MB skill, one per (chunk, term) pair.
At the shipped defaults, a 1 MB skill
Footprint composition moves from index-dominated to embedding-dominated:
Coalesced index postings
Same index — the same 3,026 terms carrying the same 36,046 references, checked through
context_index_ref_hashes, the helper the retrieve path itself uses.context_indexgoes 13.6 MB → 1.8 MB.Two things gated this, and both were real:
record_ref_hashread every singular identity field and neverref_hashes, so a posting with two refs had no identity. Fixed in A posting carrying several refs had no identity, and serving dropped it #515 — which is what makes this change possible now.MAX_SECONDARY_INDEX_REFS_PER_POSTING = 512, whichcompact_context_index_postingsobserves. A cap one producer of a record type respects and another ignores is not a bound. It now splits at the cap, stampsposting_part, and carries the singularref_hashonly for a single-ref part — matching the compactor exactly.Vector compaction was 44% of emission
1,690,624
round()calls per document, one per dimension of every chunk vector. The arithmetic is trivial; the Python loop is not. Emission 1,143 ms → 525 ms (2.18×).numpy is not assumed — the loop remains as the fallback, and both paths are covered by one equality test over exact
.5boundaries, all-zero vectors, negative zero and the empty vector, which is where two rounding implementations diverge if either does not round half to even.Index priority was a 20-way prefix scan
525,018
startswithcalls across the 36,322 terms a document emits. Every prefix is exactly"<kind>:"and every term exactlyf"{kind}:{value}", so the scan was an equality test on the kind. 40 ms → 9 ms per 40,000 terms, verified identical on 107 terms including values containing colons and kinds that merely share an opening substring with a listed one.cosine() was not a cosine
It returned a bare dot product over assumed-unit inputs, and
normalized_dense_scoreclamps(v + 1) / 2into[0, 1]. Correct only while every stored vector is unit — and every compaction option produces one that is not. It now divides by both norms, a no-op for what is stored today (a unit vector's dot already is its cosine).scale=1e5 becomes the default; int8 stays off
Measured over 500 real chunks, six EN/CN queries, e5-large @512, against the float ranking:
A uniform scale multiplies every vector by the same constant, so no pair can change places. int8 divides each vector by its own peak, so two stored vectors are scaled differently and can swap against one query — normalising the scorer took int8 from unusable to merely wrong, and #419's conclusion stands. A synthetic near-neighbour set showed int8 exact under a true cosine; it was simply too easy, which is the trap #419 documented.
Gates removed
Three that default ON and whose off-branch no test executes: group commit, upsert delta emission, and the relaxed WAL parent-dir fsync. The last was already unconditional — it read
!TS_WAL_LEGACY_RECOVERY || group_commit_enabled(), and group commit defaulted on, so the disjunction was always true.Testing
test_ingest_envelope_schemafails on this environment'sjsonschema(noDraft202012Validator), unrelated to this change.