Repository navigation
feat(dream): memory_duplicate findings for near-duplicate memory pairs - #1484
Merged
Merged
Conversation
Contributor
SCIP oracle — resolution reportHeuristic→compiler edge resolution per corpus. Δ compares resolved-after to the
resolved = |
Contributor
|
| Project | rag-rat |
| Branch | feat/dream-duplicate-findings |
| Testbed | hetzner-bigmem |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
- LL Hits (hits)
- Instructions (instructions)
- Total read+write (reads/writes)
- RAM Hits (hits)
- L1 Hits (hits)
- Estimated Cycles (cycles)
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | Estimated Cycles | cycles x 1e6 | Instructions | instructions x 1e6 | L1 Hits | hits x 1e6 | LL Hits | hits x 1e6 | RAM Hits | hits x 1e3 | Total read+write | reads/writes x 1e6 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rag_pipeline::pipeline::index cargo_resolver:resolver_config() | 📈 view plot | 1,927.03 x 1e6 | 📈 view plot | 1,207.58 x 1e6 | 📈 view plot | 1,687.82 x 1e6 | 📈 view plot | 38.76 x 1e6 | 📈 view plot | 1,296.97 x 1e3 | 📈 view plot | 1,727.88 x 1e6 |
| rag_pipeline::pipeline::query_cold cargo_resolver:resolver_built_config() | 📈 view plot | 266.29 x 1e6 | 📈 view plot | 176.29 x 1e6 | 📈 view plot | 250.84 x 1e6 | 📈 view plot | 2.86 x 1e6 | 📈 view plot | 32.16 x 1e3 | 📈 view plot | 253.74 x 1e6 |
| rag_pipeline::pipeline::query_warm cargo_resolver:resolver_index() | 📈 view plot | 238.26 x 1e6 | 📈 view plot | 157.31 x 1e6 | 📈 view plot | 224.42 x 1e6 | 📈 view plot | 2.55 x 1e6 | 📈 view plot | 30.68 x 1e3 | 📈 view plot | 227.00 x 1e6 |
Contributor
|
| Project | rag-rat |
| Branch | feat/dream-duplicate-findings |
| Testbed | hetzner-bigmem |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result seconds (s) (Result Δ%) | Upper Boundary seconds (s) (Limit %) |
|---|---|---|---|
| index_time/full_rebuild_cargo | 📈 view plot 🚷 view threshold | 6.66 s(+8.08%)Baseline: 6.16 s | 7.20 s (92.49%) |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1445. This is the dream half; the write-time warning landed in #1478.
dreamnow reports amemory_duplicatefinding for each pair of live memories whose vectors sit close enough to be the same note under other words. It uses the per-modelnear_duplicate_permillethreshold thatmemory_create'ssimilar_memoriesuses (jina-code 0.86 and the hash fallback; other models are not measured and skip the check). The reviewer merges the two and retires one, or dismisses the pair when both rules are needed.Design
ai::near_duplicate_pairs: all pairs of the active repo's embedded live memories, unit-normalized once) and passes them torag_rat_dreamasNearDuplicates { pairs, compared }.dream_run_with_passesgains a trailing parameter for them.memory_duplicatejoins the resolve sweep only when pairs were supplied.must/shouldand identifiers to separate restatements from conflicts. That was implemented and measured on 816 real memories: every one of the 13 flagged pairs "differed", restatements included. Paraphrases name different symbols and cite different details, so the label carried no information and was removed. The module doc records this.Measurements
The same live corpus (816 memories,
jinaai/jina-embeddings-v2-base-code):ponytail:note records the O(n²) ceiling and the upgrade path.Surfaces
dream-reviewskill (both copies): amemory_duplicatesection — read both, merge and retire, or dismiss.docs/mcp-tools.mdand the MCPdreamdescription list the new kind.Tests
schema_bootstrap_tests/repo_memory/duplicate_findings.rscovers:The kind-token test covers
memory_duplicate. Each test fails under a mutation of the code it guards.