Skip to content

Retrieval scored every re-summarised node on its first summary - #532

Merged
bjmeetsfo merged 1 commit into
mainfrom
fix/newest-summary-vector
Aug 31, 2026
Merged

Retrieval scored every re-summarised node on its first summary#532
bjmeetsfo merged 1 commit into
mainfrom
fix/newest-summary-vector

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Retrieval scored every re-summarised node on its first summary vector, not its current one.

The defect

ContextQuerySummaryVectors asked for one summary and took it from the front of the range:

load_context_summaries(cache, page_store, shard_id, shard, &object_key, as_of_ms, Some(1))
    .into_iter()
    .next()

load_context_summaries walks series.range(0..context_timeline_end(as_of_ms)).take(limit), and
the series is keyed by

context_timeline_key(timestamp_ms, disambiguator) = timestamp_ms * FANOUT + disambiguator % FANOUT

which ascends with time. So the first entry in the range is the oldest summary in the window,
not the newest. The arm's own comment above the call said "Only the newest summary at or before
as_of_ms is consulted", and the module comment on the neighbouring range helper says "oldest
first" — the two disagreed, and the code followed the second.

ContextUpsertSummary keys the series by valid_from_ms * FANOUT + level, so a node re-summarised
at a later time accumulates a new entry rather than replacing the old one. Every node that has
ever been re-summarised was therefore scored on a superseded embedding.

Why nothing surfaced it

A stale vector is the same width as a current one, so it produces a perfectly ordinary cosine.
There is no length mismatch, no decode failure, no log line — the only symptom is that retrieval
quality drifts toward each node's first-ever summary as the corpus matures, which looks like the
model being mediocre rather than like a bug.

The fix

load_newest_context_summary walks the same range backwards and returns the first entry that
satisfies valid_from_ms <= as_of_ms. In the ordinary case that decodes exactly one record — the
same cost as before, with the right answer.

load_latest_context_summary already returned the correct summary, but it does so by decoding the
node's whole window and taking a max_by_key, so it was not the right thing to call from a path
that runs once per candidate. load_context_summaries keeps its existing take-from-the-front
semantics, because ContextQuerySummaries is a windowed listing where that is what a limit should
mean.

Tests

a_resummarised_node_scores_on_its_newest_vector_not_its_first writes two versions of one node's
summary with deliberately opposite vectors, so choosing the wrong one cannot be mistaken for a
rounding difference. It asserts two things that pull against each other:

assertion what it stops
as of 10 000 ms, the revised vector comes back the original oldest-first behaviour
as of 1 500 ms, the first vector comes back "newest" quietly degenerating into "last written", with as_of_ms decorative

Both were checked against deliberately broken code, because a test that passes on correct code has
not yet been shown able to fail:

broken version result
.rev() dropped — the original oldest-first behaviour fails, as it must
range unbounded (0..u64::MAX), alone passes
valid_from_ms <= as_of_ms filter dropped, alone passes
both of the above together fails, as it must
real code passes

The two middle rows are worth keeping rather than tidying away. Neither fails alone because
as_of_ms is enforced twice over: walking backwards, the range bound excludes a not-yet-valid
entry, and so does the filter, so removing either leaves the other doing the job. Only removing
both leaves the constraint unenforced, and then the assertion fires.

That is a fact about the code, not a weakness in the test — but it is only knowable by running the
mutations. Taken one at a time they would have read as "this assertion guards nothing", which is
exactly the wrong conclusion to draw about a redundant safety property.

How it was found, and the instrument that found it

This also adds a counting global allocator, because attributing it is what surfaced the bug. Per
additional retrieve candidate the cost is 32.5 allocations, split:

stage allocations per extra candidate
node fetch (whole ContextNode per candidate) 12.0
summary-vector pass 19.0
everything else 1.4

The call that returns the least data — just (node_hash, vector) — was costing the most, which is
what made me read the arm.

RSS cannot substitute for this: it conflates allocated-and-held, freed-but-retained and non-heap
memory, and 71% of this system's proxy resident memory measured as allocator retention rather than
live data. Counts also do not move with machine load, which matters on a host that sits between
load 5 and 30.

It is behind the alloc-probe feature and off by default, which is not incidental. Installed
unconditionally in test builds, a wrapper adding two atomics to every allocation in the process
turned socket-bound proxy and raft tests red in a full single-threaded run — tests that pass
individually, as a group, and in a run without it. An instrument that changes the outcome of the
suite verifying the change is worse than no instrument.

The gate then creates its own trap: with the feature off every counter reads zero, and the harness
would print a tidy table of zeros that looks exactly like "this path allocates nothing". So the
harness allocates something known first and asserts the counter moved, and the probe's own
self-tests are gated on the feature too rather than asserting a property the default build
deliberately does not have.

ContextQuerySummaryVectors asked load_context_summaries for Some(1) and took the
front of the range. The summary series is keyed by context_timeline_key, which is
timestamp_ms * FANOUT + disambiguator and so ascends with time, meaning the front
of the range is the OLDEST summary in the window, not the newest.

ContextUpsertSummary keys the series by valid_from_ms * FANOUT + level, so a node
re-summarised at a later time accumulates an entry rather than replacing one.
Every node that had been re-summarised was therefore scored on a superseded
embedding -- and a stale vector is the same width as a current one, so it yields
an ordinary cosine with no error, no decode failure and nothing in the logs.

load_newest_context_summary walks the same range backwards and returns the first
entry satisfying valid_from_ms <= as_of_ms, decoding one record in the ordinary
case. load_latest_context_summary was already correct but decodes the node's
whole window, which is wrong for a path that runs once per candidate.
load_context_summaries keeps its take-from-the-front semantics for
ContextQuerySummaries, where a limit on a windowed listing should mean that.

Also adds a counting global allocator behind the alloc-probe feature, off by
default. Process RSS conflates allocated-and-held, freed-but-retained and
non-heap memory -- 71 percent of the proxy's resident memory measured as
allocator retention -- so a request-level memory change cannot be read off RSS.
Counting the calls is how this bug was found: per additional retrieve candidate
the cost is 32.5 allocations, split 12.0 for fetching whole nodes against 19.0
for the summary-vector pass, and the call returning the least data costing the
most is what prompted reading it.

Feature-gated rather than always-on in tests, and that is not incidental: two
atomics on every allocation in the process turned socket-bound proxy and raft
tests red in a full single-threaded run -- tests that pass individually, as a
group, and in a run without it. An instrument that changes the outcome of the
suite verifying the change is worse than no instrument. The gate then creates
its own trap, since with the feature off every counter reads zero and the
harness would print a table of zeros that reads as 'this path allocates
nothing', so it allocates something known first and asserts the counter moved,
and the probe's self-tests are gated too rather than asserting a property the
default build deliberately does not have.

Mutation-tested. Dropping the .rev() reproduces the original bug and fails.
Unbounding the range and dropping the valid_from_ms filter each PASS alone,
because as_of_ms is enforced twice over; removing both fails.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 31, 2026 20:17
@bjmeetsfo
bjmeetsfo merged commit 8185b64 into main Aug 31, 2026
7 checks passed
@bjmeetsfo
bjmeetsfo deleted the fix/newest-summary-vector branch August 31, 2026 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant