Repository navigation
Add agent memory benchmark #362
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
tylerhutcherson
merged 25 commits into
redis:main
from
ichko:add-agent-memory-benchmark
Sep 15, 2026
Merged
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
b27ce7f
Add the LongMemEval harness under agent-memory-benchmark/.
ichko d13fe8a
Present Redis Agent Memory, V0, and the LongMemEval harness together.
ichko 6fbc971
Give each memory adapter its own module.
ichko a4aa659
Drop the Mastra OM and Emergence Fast placeholders.
ichko c7efde1
Add a Redis Agent Memory store for the LongMemEval harness.
ichko cc98430
Fix adapter SDK mismatches and runner isolation bugs.
ichko 0572db9
Fix judge formatting, bounded Redis cleanup, and Bedrock reset.
ichko 5076c0b
Wait for stable extraction and keep failed cleanup retryable.
ichko 359fcfd
Lead the landing copy and store registry with Redis Agent Memory.
ichko 48fce67
Page memory listings and pass session dates into adapters.
ichko c8adb99
List memories without dummy search and keep digit IDs as strings.
ichko eafa48e
improve readmes and some docs
tylerhutcherson d217f65
Stream LongMemEval downloads and fail incomplete extraction waits.
ichko 0fa60ad
Merge branch 'add-agent-memory-benchmark' of https://github.com/ichko…
ichko 70bc8f5
Surface Redis long-term memory delete failures.
ichko f36ede1
Lengthen extraction wait and put session dates into Mem0 context.
ichko 50e77c2
Match provider SDK argument names and keep resume duration.
ichko 82d3af5
Skip inherited extraction waits and ingest session dates.
ichko 8cc8460
Poll Zep and Bedrock until extraction is stable.
ichko 7cd363c
Handle empty Graphiti edge groups.
ichko d7463e9
Fix harness lint so GitHub CI can pass.
ichko af09623
Scope Vertex list and reset to the example user.
ichko 92b7be7
Delete Oracle ingest threads on reset.
ichko 8959016
Restore the original LongMemEval judge, answer prompt, and hypothesis…
ichko e25f7cc
Lead the landing README with the Redis Agent Memory product.
ichko File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| name: Agent Memory Benchmark | ||
|
|
||
| on: | ||
| push: | ||
| pull_request: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| test: | ||
| runs-on: ubuntu-latest | ||
| defaults: | ||
| run: | ||
| working-directory: agent-memory-benchmark | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: astral-sh/setup-uv@v6 | ||
| with: | ||
| enable-cache: true | ||
| cache-dependency-glob: agent-memory-benchmark/uv.lock | ||
| - name: Install Python | ||
| run: uv python install 3.11 | ||
| - name: Install dependencies | ||
| run: uv sync --locked --group dev | ||
| - name: Lint | ||
| run: uv run ruff check . | ||
| - name: Check formatting | ||
| run: uv run ruff format --check . | ||
| - name: Test | ||
| run: uv run pytest | ||
| - name: Build package | ||
| run: uv build |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,23 @@ | ||
| # CLAUDE.md - Redis Agent Memory Server Project Context | ||
| # CLAUDE.md - Redis Agent Memory Project Context | ||
|
|
||
| This project lives in [`V0/`](./V0/). See [`V0/AGENTS.md`](./V0/AGENTS.md) for | ||
| full instructions (and [`V0/CLAUDE.md`](./V0/CLAUDE.md)). Run commands from | ||
| inside `V0/`. | ||
| Read [`AGENTS.md`](./AGENTS.md) first. It maps the two projects in this | ||
| repository: [`V0/`](./V0/) (the Agent Memory Server) and | ||
| [`agent-memory-benchmark/`](./agent-memory-benchmark/) (the LongMemEval | ||
| harness). | ||
|
|
||
| For the server, see [`V0/AGENTS.md`](./V0/AGENTS.md) and | ||
| [`V0/CLAUDE.md`](./V0/CLAUDE.md). For the harness, see | ||
| [`agent-memory-benchmark/README.md`](./agent-memory-benchmark/README.md). | ||
|
|
||
| Run commands from inside the project directory, not from the repository root. | ||
|
|
||
| Keep the repository narrative intact when changing landing or project context: | ||
|
|
||
| - `V0/` is intentionally preserved as the pivotal open research foundation that | ||
| informed Redis Agent Memory productization, not as the supported production | ||
| distribution. | ||
| - Redis Agent Memory in Redis Iris is the official product path; keep its | ||
| create-service and SDK/API quickstarts easy to find. | ||
| - The benchmark is an open audit and reproducibility surface, not a timeless | ||
| vendor ranking. Benchmark claims must name the provider configuration, split, | ||
| models, retrieval settings, artifacts, and run date. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,50 +1,57 @@ | ||
| <div align=center> | ||
|
|
||
| # Redis Agent Memory | ||
|
|
||
| A memory layer that gives agents intelligent short-term memory and persistent context across conversations. | ||
| [Redis Agent Memory](https://redis.io/agent-memory/) is Redis's managed memory layer for AI agents: a persistent store that lets an agent recall facts, events, and preferences across sessions, without stuffing every token of history back into the prompt. It is part of [Redis Iris](https://redis.io/iris/), Redis's real-time context engine for agents. | ||
|
|
||
| </div> | ||
| [Redis Agent Memory docs](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/) · [Benchmark harness](./agent-memory-benchmark/) · [Benchmark write-up](https://redis.github.io/redis-ai-research-public/longmemeval-agent-memory/) | ||
|
|
||
| ## Redis Agent Memory in Redis Iris | ||
| ## Getting started | ||
|
|
||
| [Redis Agent Memory in Redis Iris](https://redis.io/agent-memory/) is Redis’s official managed path for teams that want agent memory as a service, not another subsystem to build and operate themselves. [Redis Iris](https://redis.io/iris/) is the real-time context engine for agents, designed to deliver fresh, relevant context at runtime, and Redis Agent Memory is the part of Iris that makes context compound across turns, sessions, channels, and agents. | ||
| - Product overview: [Redis Iris](https://redis.io/iris/) | ||
| - Agent Memory overview: [Redis Agent Memory docs](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/) | ||
| - Redis Cloud service guide: [Redis Agent Memory on Redis Cloud](https://redis.io/docs/latest/operate/rc/context-engine/agent-memory/) | ||
|
|
||
| Redis Agent Memory in Iris gives you the Redis-managed experience: a persistent, structured memory layer for AI agents exposed through a REST API and client libraries, with dedicated endpoints, secure API key management, configurable memory schemas, and automatic TTL-based lifecycle management. The point is not just storage. It is to remove the custom memory infrastructure teams otherwise end up building around session handling, extraction, retrieval, and lifecycle management. | ||
| On Redis Cloud: | ||
|
|
||
| Redis Agent Memory uses a two-tier model. Session memory keeps the active conversation state, session history, and session-specific metadata close at hand, with configurable TTL control for retention. Long-term memory stores extracted facts and learned patterns from past interactions as text plus vector embeddings for semantic retrieval. As new events are written to working memory, Redis Agent Memory automatically extracts important information and promotes it to long-term memory in the background, so memory accumulates without slowing down the live agent loop. | ||
| 1. [Create a database](https://redis.io/docs/latest/operate/rc/databases/create-database/). | ||
| 2. [Create an Agent Memory service](https://redis.io/docs/latest/operate/rc/context-engine/agent-memory/create-service/). | ||
| 3. Use the [Python SDK](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/python-sdk-quickstart/), [TypeScript SDK](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/typescript-sdk-quickstart/), or [REST API](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/rest-api-quickstart/) from your app. | ||
| 4. Manage the service with the [Redis Cloud guide](https://redis.io/docs/latest/operate/rc/context-engine/agent-memory/). | ||
|
|
||
| That matters because Redis Iris is not just a memory feature in isolation. It is a broader context engine built to address the production problems agents actually hit: fragmented data, stale operational state, slow retrieval, and interactions that do not improve over time. Within that story, Redis Agent Memory is the compounding memory layer; [Redis Context Retriever](https://redis.io/context-retriever/) makes business data navigable; [Redis Data Integration](https://redis.io/data-integration/) keeps operational state fresh; and [Redis LangCache](https://redis.io/langcache/) helps repeated work stay inside the latency budget. | ||
| Implementation details: [API and SDK examples](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/) and the [API reference](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/api-reference/). | ||
|
|
||
| If you are evaluating the supported Redis path, these are the best places to start: | ||
| ## How it works | ||
|
|
||
| - Product overview: [Redis Iris](https://redis.io/iris/) | ||
| - Agent Memory overview: [Redis Agent Memory docs](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/) | ||
| - Redis Cloud service guide: [Redis Agent Memory on Redis Cloud](https://redis.io/docs/latest/operate/rc/context-engine/agent-memory/) | ||
| Redis Agent Memory uses a two-tier model. Session memory keeps active conversation state close at hand, with configurable TTL. Long-term memory stores extracted facts from past sessions as text plus vector embeddings for later retrieval. | ||
|
|
||
| As new events land in session memory, Redis Agent Memory extracts what matters and promotes it to long-term memory in the background. Memory can accumulate without slowing the live agent loop. | ||
|
|
||
| Keeping full conversation history in the prompt does not scale. Context windows are bounded, long prompts are slow and expensive, and accuracy degrades as prompts grow. Agent memory lets an agent stay fast and cheap on every turn while still recalling what happened weeks ago. | ||
|
|
||
| A practical getting-started flow on Redis Cloud looks like this: | ||
| ## Validated independently | ||
|
|
||
| - [Create a database](https://redis.io/docs/latest/operate/rc/databases/create-database/) | ||
| - [Create an Agent Memory service](https://redis.io/docs/latest/operate/rc/context-engine/agent-memory/create-service/) | ||
| - [Use the Agent Memory API](https://redis.io/docs/latest/operate/rc/context-engine/agent-memory/use-agent-memory/) from your application | ||
| - [View and manage your service](https://redis.io/docs/latest/operate/rc/context-engine/agent-memory/view-service/) | ||
| We ran a best-effort LongMemEval comparison across a sweep of memory systems, on a shared answer model and judge, to put those systems on more equal footing (Wu et al., ICLR 2025). Some vendors publish higher scores than we report. We were not able to reproduce those numbers under this protocol. | ||
|
|
||
| For implementation details and usage examples, see: | ||
| A separate research configuration that combined two retrieval strategies (Instruct extraction and Remis) reached **86.5%** task-averaged accuracy in our evaluation, at about $0.07 per session. Remis is **not** in Redis Agent Memory in Iris yet, and that configuration is **not** what this public harness runs. Do not treat 86.5% as a Redis Cloud product score. | ||
|
|
||
| - [API and SDK examples](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/api-examples/) | ||
| - [API reference](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/api-reference/) | ||
| The public harness in [`agent-memory-benchmark/`](./agent-memory-benchmark/) evaluates the shipping Redis Agent Memory REST API and other adapters. It is an audit surface, not a timeless ranking. A result is evidence about a specific provider version, split, models, retrieval settings, artifacts, and run date. | ||
|
|
||
| ```bash | ||
| git clone https://github.com/redis/agent-memory-server.git | ||
| cd agent-memory-server/agent-memory-benchmark | ||
| ``` | ||
|
|
||
| Setup, protocol, and provider recipes: [`agent-memory-benchmark/README.md`](./agent-memory-benchmark/README.md). | ||
|
|
||
| ## V0 — the open-source research foundation | ||
|
|
||
| [**`V0/`**](./V0/) contains the original Redis Agent Memory Server: an open-source reference implementation for agent memory with REST and MCP interfaces, working and long-term memory, configurable extraction strategies, and Redis-backed semantic search. | ||
| It serves as the research foundation and architectural starting point for Redis Agent Memory, but it is not the current supported production path. | ||
| [`V0/`](./V0/) is the original Redis Agent Memory Server. It informed the Iris product. We keep it here as an open research artifact, not as the supported production distribution. Start at [`V0/README.md`](./V0/README.md). | ||
|
|
||
| ## Citing this work | ||
|
|
||
| Redis Applied AI Research, "How Redis Agent Memory Stacks Up: Benchmarking Memory Systems for AI Agents on LongMemEval," 2026. | ||
|
|
||
| - **Start here:** [`V0/README.md`](./V0/README.md) | ||
| - **Documentation:** [`V0/docs/`](./V0/docs/index.md) | ||
| - Build, test, and run everything from inside `V0/` (e.g. `cd V0 && make test`). | ||
| Also see [Building and evaluating long-term conversational memory](https://redis.github.io/redis-ai-research-public/longmemeval-agent-memory/). | ||
|
|
||
| ## License | ||
|
|
||
| This project is licensed under the **Apache License 2.0** (Redis, Inc.). See | ||
| [`LICENSE`](./LICENSE) at the repository root. | ||
| Apache License 2.0 (Redis, Inc.). See [`LICENSE`](./LICENSE). | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| # Answer generation and LongMemEval judging | ||
| OPENAI_API_KEY= | ||
|
|
||
| # Public SaaS providers | ||
| MEM0_API_KEY= | ||
| ZEP_API_KEY= | ||
| SUPERMEMORY_API_KEY= | ||
|
|
||
| # Redis Agent Memory | ||
| REDIS_AGENT_MEMORY_URL= | ||
| REDIS_AGENT_MEMORY_STORE_ID= | ||
| REDIS_AGENT_MEMORY_API_KEY= | ||
|
|
||
| # Graphiti / Neo4j | ||
| NEO4J_URI=bolt://localhost:7687 | ||
| NEO4J_USER=neo4j | ||
| NEO4J_PASSWORD= | ||
|
|
||
| # Google Vertex AI Memory Bank | ||
| GOOGLE_CLOUD_PROJECT= | ||
| GOOGLE_CLOUD_LOCATION=us-central1 | ||
|
|
||
| # AWS Bedrock AgentCore Memory | ||
| AWS_REGION=us-east-1 | ||
|
|
||
| # Oracle Agent Memory | ||
| ORACLE_MEMORY_DB_USER= | ||
| ORACLE_MEMORY_DB_PASSWORD= | ||
| ORACLE_MEMORY_DB_CONNECT_STRING= |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| .DS_Store | ||
| .env | ||
| .env.* | ||
| !.env.example | ||
| .python-version | ||
| !.python-version | ||
| .venv/ | ||
| __pycache__/ | ||
| *.egg-info/ | ||
| *.py[cod] | ||
| .coverage | ||
| coverage.xml | ||
| htmlcov/ | ||
| .pytest_cache/ | ||
| .ruff_cache/ | ||
| .mypy_cache/ | ||
| dist/ | ||
| build/ | ||
|
|
||
| # Downloaded datasets and resumable benchmark output | ||
| data/ | ||
| experiment_results/ | ||
| *.jsonl | ||
| *.log | ||
|
|
||
| # Leftover Agent Memory Server checkout (do not publish) | ||
| agent_memory_server/ | ||
| agent-memory-client/ | ||
|
|
||
| # Provider and cloud state | ||
| .aws/ | ||
| .gcloud/ | ||
| .terraform/ | ||
| terraform.tfstate* | ||
| cdk.out/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| repos: | ||
| - repo: https://github.com/astral-sh/ruff-pre-commit | ||
| rev: v0.14.8 | ||
| hooks: | ||
| - id: ruff-check | ||
| args: [--fix] | ||
| - id: ruff-format | ||
| - repo: https://github.com/pre-commit/pre-commit-hooks | ||
| rev: v6.0.0 | ||
| hooks: | ||
| - id: check-added-large-files | ||
| - id: check-merge-conflict | ||
| - id: check-toml | ||
| - id: check-yaml | ||
| - id: detect-private-key | ||
| - id: end-of-file-fixer | ||
| - id: trailing-whitespace |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 3.11 |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.