Repository navigation
feat: add describe recall, the shape of what a recall reaches - #628
RonsenbergVI wants to merge 5 commits into
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
🔴 RegressionCompared with the newest nightly run of Failedinternal/query · allocations do not grow
Tip To accept a regression worth what it costs, label the pull request SlowerBy 10% or more and significant. Not a failure: the baseline was timed on another runner. internal/graph
internal/query
Not run, nothing they build on changed: internal/graph ·
|
| benchmark | unit | baseline | this run | vs base | |
|---|---|---|---|---|---|
| Search/depth0 | sec/op | 15.3 ms | 15.08 ms | ~ | p=0.796 |
| Search/depth1 | sec/op | 15.23 ms | 15.75 ms | +3.42% | p=0.023 |
| Search/depth2 | sec/op | 14.91 ms | 16.24 ms | +8.98% | p=0.000 |
| Search/vector | sec/op | 13.09 ms | 15.21 ms | +16.26% | p=0.000 |
| Search/conversation | sec/op | 16.43 ms | 18.05 ms | +9.84% | p=0.000 |
| Search/depth0 | B/op | 6.22 MiB | 6.22 MiB | +0.00% | p=0.000 |
| Search/depth1 | B/op | 6.399 MiB | 6.399 MiB | +0.00% | p=0.000 |
| Search/depth2 | B/op | 6.399 MiB | 6.399 MiB | +0.00% | p=0.000 |
| Search/vector | B/op | 4.228 MiB | 4.228 MiB | +0.00% | p=0.000 |
| Search/conversation | B/op | 6.868 MiB | 6.868 MiB | +0.00% | p=0.000 |
| Search/depth0 | allocs/op | 281 | 280 | -0.36% | p=0.000 |
| Search/depth1 | allocs/op | 450 | 450 | ~ | p=1.000 |
| Search/depth2 | allocs/op | 450 | 450 | ~ | p=1.000 |
| Search/vector | allocs/op | 392 | 392 | ~ | p=1.000 |
| Search/conversation | allocs/op | 645 | 645 | ~ | p=1.000 |
internal/query · all benchmarks
| benchmark | unit | baseline | this run | vs base | |
|---|---|---|---|---|---|
| RememberCommit/size-100 | sec/op | 8.408 µs | 8.713 µs | +3.62% | p=0.000 |
| RememberCommit/size-10000 | sec/op | 8.427 µs | 8.777 µs | +4.15% | p=0.000 |
| RecallCommit/size-1000 | sec/op | 1.079 ms | 1.106 ms | +2.48% | p=0.000 |
| RecallCommit/size-10000 | sec/op | 14.12 ms | 15.59 ms | +10.44% | p=0.000 |
| RememberCommit/size-100 | B/op | 2.483 KiB | 2.499 KiB | ~ | p=0.060 |
| RememberCommit/size-10000 | B/op | 2.491 KiB | 2.518 KiB | +1.08% | p=0.000 |
| RecallCommit/size-1000 | B/op | 381.5 KiB | 381.4 KiB | -0.02% | p=0.000 |
| RecallCommit/size-10000 | B/op | 3.53 MiB | 3.532 MiB | +0.06% | p=0.000 |
| RememberCommit/size-100 | allocs/op | 38 | 38 | ~ | p=1.000 |
| RememberCommit/size-10000 | allocs/op | 38 | 38 | ~ | p=1.000 |
| RecallCommit/size-1000 | allocs/op | 103 | 103 | ~ | p=0.124 |
| RecallCommit/size-10000 | allocs/op | 443 | 466.5 | +5.30% | p=0.000 |
pkg/server · BenchmarkRetrievalQuality
| benchmark | unit | baseline | this run | vs base | |
|---|---|---|---|---|---|
| RetrievalQuality/text/all | sec/op | 197.5 ms | 209.1 ms | ~ | p=1.000 |
| RetrievalQuality/text/multi-hop | sec/op | 31.31 ms | 34.47 ms | ~ | p=1.000 |
| RetrievalQuality/text/temporal | sec/op | 36.36 ms | 39.13 ms | ~ | p=1.000 |
| RetrievalQuality/text/open-domain | sec/op | 12.18 ms | 11.88 ms | ~ | p=1.000 |
| RetrievalQuality/text/single-hop | sec/op | 69.95 ms | 73.62 ms | ~ | p=1.000 |
| RetrievalQuality/text/adversarial | sec/op | 46.52 ms | 52.01 ms | ~ | p=1.000 |
| RetrievalQuality/vectors/all | sec/op | 272.7 ms | 289.2 ms | ~ | p=1.000 |
| RetrievalQuality/vectors/multi-hop | sec/op | 43.11 ms | 47.49 ms | ~ | p=1.000 |
| RetrievalQuality/vectors/temporal | sec/op | 50.93 ms | 54.73 ms | ~ | p=1.000 |
| RetrievalQuality/vectors/open-domain | sec/op | 16.07 ms | 16.96 ms | ~ | p=1.000 |
| RetrievalQuality/vectors/single-hop | sec/op | 96.27 ms | 105.6 ms | ~ | p=1.000 |
| RetrievalQuality/vectors/adversarial | sec/op | 64.31 ms | 71.25 ms | ~ | p=1.000 |
| RetrievalQuality/text/all | turn-f1@10 | 0.1512 | 0.1512 | 0.00% | exact |
| RetrievalQuality/text/multi-hop | turn-f1@10 | 0.1059 | 0.1059 | 0.00% | exact |
| RetrievalQuality/text/temporal | turn-f1@10 | 0.1671 | 0.1671 | 0.00% | exact |
| RetrievalQuality/text/open-domain | turn-f1@10 | 0.09091 | 0.09091 | 0.00% | exact |
| RetrievalQuality/text/single-hop | turn-f1@10 | 0.158 | 0.158 | 0.00% | exact |
| RetrievalQuality/text/adversarial | turn-f1@10 | 0.1734 | 0.1734 | 0.00% | exact |
| RetrievalQuality/vectors/all | turn-f1@10 | 0.1666 | 0.1666 | 0.00% | exact |
| RetrievalQuality/vectors/multi-hop | turn-f1@10 | 0.1506 | 0.1506 | 0.00% | exact |
| RetrievalQuality/vectors/temporal | turn-f1@10 | 0.1818 | 0.1818 | 0.00% | exact |
| RetrievalQuality/vectors/open-domain | turn-f1@10 | 0.1543 | 0.1543 | 0.00% | exact |
| RetrievalQuality/vectors/single-hop | turn-f1@10 | 0.1606 | 0.1606 | 0.00% | exact |
| RetrievalQuality/vectors/adversarial | turn-f1@10 | 0.1773 | 0.1773 | 0.00% | exact |
| RetrievalQuality/text/all | turn-p@1 | 0.3299 | 0.3299 | 0.00% | exact |
| RetrievalQuality/text/multi-hop | turn-p@1 | 0.125 | 0.125 | 0.00% | exact |
| RetrievalQuality/text/temporal | turn-p@1 | 0.5946 | 0.5946 | 0.00% | exact |
| RetrievalQuality/text/open-domain | turn-p@1 | 0.09091 | 0.09091 | 0.00% | exact |
| RetrievalQuality/text/single-hop | turn-p@1 | 0.3286 | 0.3286 | 0.00% | exact |
| RetrievalQuality/text/adversarial | turn-p@1 | 0.3191 | 0.3191 | 0.00% | exact |
| RetrievalQuality/vectors/all | turn-p@1 | 0.3858 | 0.3858 | 0.00% | exact |
| RetrievalQuality/vectors/multi-hop | turn-p@1 | 0.25 | 0.25 | 0.00% | exact |
| RetrievalQuality/vectors/temporal | turn-p@1 | 0.6486 | 0.6486 | 0.00% | exact |
| RetrievalQuality/vectors/open-domain | turn-p@1 | 0.09091 | 0.09091 | 0.00% | exact |
| RetrievalQuality/vectors/single-hop | turn-p@1 | 0.3857 | 0.3857 | 0.00% | exact |
| RetrievalQuality/vectors/adversarial | turn-p@1 | 0.3404 | 0.3404 | 0.00% | exact |
| RetrievalQuality/text/all | turn-recall@10 | 0.7652 | 0.7652 | 0.00% | exact |
| RetrievalQuality/text/multi-hop | turn-recall@10 | 0.3047 | 0.3047 | 0.00% | exact |
| RetrievalQuality/text/temporal | turn-recall@10 | 0.9189 | 0.9189 | 0.00% | exact |
| RetrievalQuality/text/open-domain | turn-recall@10 | 0.2727 | 0.2727 | 0.00% | exact |
| RetrievalQuality/text/single-hop | turn-recall@10 | 0.8571 | 0.8571 | 0.00% | exact |
| RetrievalQuality/text/adversarial | turn-recall@10 | 0.9362 | 0.9362 | 0.00% | exact |
| RetrievalQuality/vectors/all | turn-recall@10 | 0.8236 | 0.8236 | 0.00% | exact |
| RetrievalQuality/vectors/multi-hop | turn-recall@10 | 0.4141 | 0.4141 | 0.00% | exact |
| RetrievalQuality/vectors/temporal | turn-recall@10 | 1 | 1 | 0.00% | exact |
| RetrievalQuality/vectors/open-domain | turn-recall@10 | 0.5455 | 0.5455 | 0.00% | exact |
| RetrievalQuality/vectors/single-hop | turn-recall@10 | 0.8714 | 0.8714 | 0.00% | exact |
| RetrievalQuality/vectors/adversarial | turn-recall@10 | 0.9574 | 0.9574 | 0.00% | exact |
| RetrievalQuality/text/all | B/op | 65.55 MiB | 65.09 MiB | ~ | p=1.000 |
| RetrievalQuality/text/multi-hop | B/op | 10.75 MiB | 10.9 MiB | ~ | p=1.000 |
| RetrievalQuality/text/temporal | B/op | 12.28 MiB | 12.52 MiB | ~ | p=1.000 |
| RetrievalQuality/text/open-domain | B/op | 3.913 MiB | 3.712 MiB | ~ | p=1.000 |
| RetrievalQuality/text/single-hop | B/op | 22.46 MiB | 22.54 MiB | ~ | p=1.000 |
| RetrievalQuality/text/adversarial | B/op | 15.39 MiB | 15.31 MiB | ~ | p=1.000 |
| RetrievalQuality/vectors/all | B/op | 91.51 MiB | 91.8 MiB | ~ | p=1.000 |
| RetrievalQuality/vectors/multi-hop | B/op | 14.96 MiB | 15.11 MiB | ~ | p=1.000 |
| RetrievalQuality/vectors/temporal | B/op | 17.37 MiB | 17.43 MiB | ~ | p=1.000 |
| RetrievalQuality/vectors/open-domain | B/op | 5.435 MiB | 5.368 MiB | ~ | p=1.000 |
| RetrievalQuality/vectors/single-hop | B/op | 31.88 MiB | 32.09 MiB | ~ | p=1.000 |
| RetrievalQuality/vectors/adversarial | B/op | 21.73 MiB | 21.62 MiB | ~ | p=1.000 |
| RetrievalQuality/text/all | allocs/op | 108,434 | 108,685 | ~ | p=1.000 |
| RetrievalQuality/text/multi-hop | allocs/op | 16,753 | 16,828 | ~ | p=1.000 |
| RetrievalQuality/text/temporal | allocs/op | 19,346 | 19,429 | ~ | p=1.000 |
| RetrievalQuality/text/open-domain | allocs/op | 6,210 | 6,108 | ~ | p=1.000 |
| RetrievalQuality/text/single-hop | allocs/op | 37,494 | 37,538 | ~ | p=1.000 |
| RetrievalQuality/text/adversarial | allocs/op | 25,146 | 25,163 | ~ | p=1.000 |
| RetrievalQuality/vectors/all | allocs/op | 204,970 | 204,999 | ~ | p=1.000 |
| RetrievalQuality/vectors/multi-hop | allocs/op | 33,012 | 33,101 | ~ | p=1.000 |
| RetrievalQuality/vectors/temporal | allocs/op | 38,201 | 38,166 | ~ | p=1.000 |
| RetrievalQuality/vectors/open-domain | allocs/op | 11,827 | 11,778 | ~ | p=1.000 |
| RetrievalQuality/vectors/single-hop | allocs/op | 70,741 | 70,710 | ~ | p=1.000 |
| RetrievalQuality/vectors/adversarial | allocs/op | 48,289 | 48,232 | ~ | p=1.000 |
39ecf0f to
9e5f93b
Compare
describe in front of a recall answers with the catalogue of every fact the recall would rank instead of the facts: how many, the span they hold on each clock, and the topics and entities they are filed under with each one's count. An empty body describes the whole graph. It runs through /api/v1/q, both MCP bridges' recall tool and the Python SDK's client.describe.
9e5f93b to
869bfce
Compare
There was a problem hiding this comment.
No issues found in the diff at 7c1f566. Re-checked the changes since 3f59030, the diff only: _read now resolves a describe's own vector with bool(embed) and its When bounds with the caller's embed, as recall does. embed=False still sends no vector at all, embed=True without an embedder still raises, and the SDK README and python-sdk.mdx now say the same as the code.
There was a problem hiding this comment.
No issues found in the diff at 37b5279. Re-checked the changes since 7c1f566, the diff only: Times.hold now ends a span a microsecond past an instant it closes on, keeps a set of one instant as that instant, and leaves Interval.Hull and so when untouched. Both renderers now print the last fact's day ((end - 1µs) in Python, End - 1ns in Go). The Go, server and SDK pins moved with it, and the HTTP API and Commands examples match the new ends.
# Conflicts: # docs/architecture/components/engine.mdx # docs/getting-started/glossary.mdx # docs/http-api/endpoints/explain.mdx # docs/http-api/endpoints/query.mdx # docs/http-api/errors.mdx # docs/integrations/mcp.mdx # docs/query-language/foundations/commands.mdx # docs/query-language/reference/grammar.mdx # docs/query-language/tokens/keywords.mdx # internal/query/doc.go # internal/query/optimisation/dedupe_test.go # internal/query/parser/ast.go # internal/query/parser/doc.go # internal/query/parser/parser.go # internal/query/parser/parser_test.go # internal/query/query.go # internal/query/query_test.go # mcp/README.md # mcp/src/fraise_mcp/server.py # mcp/src/tests/server_test.py # pkg/mcp/server.go # pkg/mcp/types.go # pkg/server/server_test.go # sdk/python/src/fraise_sdk/client.py # sdk/python/src/fraise_sdk/query.py # sdk/python/src/tests/conftest.py # sdk/python/src/tests/models_test.py # tests/e2e/parser_test.py # tests/e2e/query_cache_test.py # tests/integration/server_test.py # tests/integration/tools_test.py
Pull Request
Type of change
Description of the changes
describe recall <body>answers with the catalogue of what the recall reaches instead of its facts: how many facts, the span they hold on happened with said and written beside it, and the topics and entities they are filed under with each one's count. The set is every candidate the recall would rank, not its firsttop:. Under describe,top:caps the anchors listed andsince:/until:bound the facts counted, and an empty body describes the whole graph. It goes through/api/v1/q, the recall tool of both MCP bridges, whose description now says to describe first, and the Python SDK'sclient.describe.Motivation for the changes
Nothing read the catalogue: an agent in a fresh session had no way to learn how memory is filed before its first recall. Part of #434; its SessionStart hook item waits on #427.
Breaking change?
If yes, fill this in
What breaks:
Before → after:
Migration:
How it was tested
go test ./...,go vet ./...,gofmt -l ., golangci-lint v2.12.2: cleanmake test-py(357),make test-mcp(61), and against fresh daemons built from the tree:tests/e2e(749), the SDK's-m integrationhalf (162), fraise-mcp's (3),make test-integration(22)describe recall@0describe recall@0 since:'2023-06' until:'2023-07'describe recall@0 topic:'conv-conv-26'describe recall@0 paintingdescribe recall@0 painting topic:'conv-conv-26'Performance regression
Checklist
Notes for reviewers