Skip to content

[feat] Add cardinality-bounded semantic join execution - #11

Merged
SleepyLGod merged 5 commits into
mainfrom
codex/semantic-join-topk
Aug 31, 2026
Merged

SleepyLGod merged 5 commits into
mainfrom
codex/semantic-join-topk

Conversation

@SleepyLGod

@SleepyLGod SleepyLGod commented Aug 30, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • add exact-key and per-left top-k contracts to semantic joins
  • provide listwise and pairwise top-k physical execution with optional Search-Filter blocking
  • lower exclusive semantic grouping through partition-aware join-map maintenance
  • expose the physical method for Zep benchmark conditions and record it in provenance
  • add a separate Mem0 Search-Filter causal-audit tool

Commit structure

  1. [feat-semantic] operator, planner, adapter, API documentation, and core tests
  2. [feat-benchmark] Zep policies, benchmark CLI wiring, validation, and provenance
  3. [analysis] read-only Mem0 causal audit and tests

Current status

This PR is intentionally a draft. The implementation has passed the existing local Ruff, Pyright, full pytest, and git diff --check gates before the commits were split. Exact committed-tree CI, final documentation review, and any remaining review findings must be completed before marking it ready.

Do not merge yet.

Summary by CodeRabbit

  • New Features

    • Added exclusive semantic grouping with partitioning, labels, and configurable membership.
    • Added exact-key filtering and bounded top-k matching for semantic joins.
    • Added support for preserving matched and unmatched join records.
    • Added incremental maintenance for relational inner joins.
    • Added a read-only audit tool for validating search, retrieval, and top-k evidence.
  • Bug Fixes

    • Improved validation and error handling for unsupported grouping and join configurations.
  • Documentation

    • Updated operator and evaluation guidance for grouping, joins, and audit workflows.

- extend semantic joins with exact-key and per-left top-k contracts
- add listwise and pairwise top-k execution with optional candidate filtering
- lower exclusive semantic grouping through partition-aware join-map maintenance
- document and test the operator, membership, and physical execution contracts

Signed-off-by: Von <haodong.lu.me@gmail.com>
- configure Zep grouping with explicit exclusive membership
- expose and validate semantic join top-k methods across benchmark CLIs
- propagate the selected physical method through driver configuration and provenance
- document and test the supported join-map argument combinations

Signed-off-by: Von <haodong.lu.me@gmail.com>
- compare Native, Oracle, and Search-Filter benchmark evidence
- audit omitted Oracle-positive pairs and counterfactual retained memories
- decompose retrieval, answer, and grading score changes
- reject incomplete event-count evidence instead of assuming a full run

Signed-off-by: Von <haodong.lu.me@gmail.com>
@coderabbitai

coderabbitai Bot commented Aug 30, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

Next included review available in 36 minutes.

View limit details

Limit details: You’ve used the included review currently available.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: 43e43741-877e-4e27-9002-0b928ec6d461

📥 Commits

Reviewing files that changed from the base of the PR and between 372a501 and 2766530.

📒 Files selected for processing (2)
  • src/agent_memory/adapters/lotus/sem_join.py
  • tests/test_semantic_pair_execution.py
ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: e288fb43-c3ed-40d9-aff8-bcf4dbcd7e94

📥 Commits

Reviewing files that changed from the base of the PR and between dc4294f and 372a501.

📒 Files selected for processing (33)
  • docs/design/groupby_agg.md
  • docs/design/operator-api.md
  • src/agent_memory/adapters/lotus/adapter.py
  • src/agent_memory/adapters/lotus/context.py
  • src/agent_memory/adapters/lotus/pair_execution.py
  • src/agent_memory/adapters/lotus/relational.py
  • src/agent_memory/adapters/lotus/sem_groupby.py
  • src/agent_memory/adapters/lotus/sem_join.py
  • src/agent_memory/adapters/lotus/sem_topk_join.py
  • src/agent_memory/evaluation/agent_memory_drivers.py
  • src/agent_memory/evaluation/run.py
  • src/agent_memory/memories/zep/policy.py
  • src/agent_memory/memories/zep/policy_extended.py
  • src/agent_memory/planner/differential_policy.py
  • src/agent_memory/planner/rules.py
  • src/agent_memory/policy/relation.py
  • src/agent_memory/policy/schema.py
  • src/agent_memory/runtime/executor.py
  • tests/test_agent_memory_benchmark_drivers.py
  • tests/test_benchmark_cli.py
  • tests/test_benchmark_harness.py
  • tests/test_differential_runtime.py
  • tests/test_interface_scaffold.py
  • tests/test_mem0_search_filter_causal_audit.py
  • tests/test_relational_join_ivm.py
  • tests/test_sem_topk_join.py
  • tests/test_semantic_pair_execution.py
  • tests/test_zep_memory_policy.py
  • tools/analysis/mem0_search_filter_causal_audit.py
  • tools/evaluation/README.md
  • tools/evaluation/locomo.py
  • tools/evaluation/longmemeval.py
  • tools/evaluation/memory_agent_bench.py
💤 Files with no reviewable changes (2)
  • tests/test_benchmark_harness.py
  • tests/test_differential_runtime.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

This change adds semantic join keys and top-k matching, exclusive semantic grouping, partition-aware join-map maintenance, incremental relational inner joins, configurable Lotus resolvers, site-specific semantic contracts, and a read-only Mem0 causal-audit CLI.

Changes

Semantic joins and grouped aggregation

Layer / File(s) Summary
Semantic contracts and execution
docs/design/*, src/agent_memory/policy/relation.py, src/agent_memory/adapters/lotus/sem_join.py, src/agent_memory/adapters/lotus/sem_topk_join.py
Adds membership, partition_by, exact join keys, bounded matches, and listwise, pairwise, and proxy-only execution paths.
Join-map planner and profiles
src/agent_memory/planner/rules.py, src/agent_memory/adapters/lotus/pair_execution.py, src/agent_memory/evaluation/*, src/agent_memory/memories/zep/*
Adds partition-restricted join-map maintenance, endpoint-specific semantic profiles, exclusive Zep grouping, and top-k configuration propagation.
Semantic validation and CLI coverage
tests/test_interface_scaffold.py, tests/test_sem_topk_join.py, tests/test_agent_memory_benchmark_drivers.py, tests/test_benchmark_cli.py, tools/evaluation/*, tools/evaluation/README.md
Validates planner output, candidate limits, provenance, profile discovery, CLI restrictions, and removal of prefer-join-map.

Relational incremental maintenance

Layer / File(s) Summary
Relational state and let execution
src/agent_memory/planner/differential_policy.py, src/agent_memory/runtime/executor.py, src/agent_memory/adapters/lotus/relational.py, src/agent_memory/adapters/lotus/adapter.py, src/agent_memory/policy/schema.py
Adds append-only maintenance for inner joins, relational-state execution, and single-evaluation let bindings.
Relational validation
tests/test_relational_join_ivm.py, tests/test_interface_scaffold.py
Covers multiplicity, self-joins, predicates, filters, retractions, snapshots, and non-inner fallback behavior.

Mem0 causal audit

Layer / File(s) Summary
Evidence loading and validation
tools/analysis/mem0_search_filter_causal_audit.py, tests/test_mem0_search_filter_causal_audit.py
Adds directory and tar readers, typed evidence records, artifact validation, digests, and synthetic fixtures.
Analysis and reporting
tools/analysis/mem0_search_filter_causal_audit.py, tests/test_mem0_search_filter_causal_audit.py
Adds condition comparison, top-k pair auditing, scorer parity, sem_filter analysis, score summaries, usage metrics, and deterministic JSON output.

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk: ⚪ Minimal · up to 372a5

The change adds bounded semantic-join execution and related benchmark tooling. No actionable merge-blocking risk remains from the supplied current-head information beyond normal checks and review.


Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/settings/billing.

Comment @coderabbitai help to get the list of available commands.

@SleepyLGod SleepyLGod self-assigned this Aug 30, 2026
- compile deterministic inner joins into bag-preserving delta maintenance
- combine old results with left, right, and cross-delta join terms
- fall back to full recomputation when parent rows are retracted
- cover self joins, multiplicity, downstream deltas, and checkpoint restore

Signed-off-by: Von <haodong.lu.me@gmail.com>
@SleepyLGod
SleepyLGod marked this pull request as ready for review August 31, 2026 05:14
Copilot AI lite review requested due to automatic review settings August 31, 2026 05:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-08-31T05:46:10.804990Z 2766530 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@SleepyLGod SleepyLGod added bug Something isn't working enhancement New feature or request feature labels Aug 31, 2026
@SleepyLGod

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 372a5010f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agent_memory/adapters/lotus/sem_join.py Outdated
- use original hashable composite values for exact semantic join partitions
- prevent string coercion from merging distinct typed keys
- cover mixed datetime/string values and Zep-style tuple endpoints

Signed-off-by: Von <haodong.lu.me@gmail.com>
@SleepyLGod

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2766530f2e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agent_memory/adapters/lotus/sem_join.py
@SleepyLGod
SleepyLGod merged commit e362190 into main Aug 31, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants