[feat-semantic] Add differential semantic dataflow execution by SleepyLGod · Pull Request #12 · SleepyLGod/agent-memory

SleepyLGod · 2026-09-11T15:31:27Z

Summary

add a public semantic-dataflow facade with scalar expressions, relational aggregates, unordered limit, and exact incremental aggregate state
derive append-only semantic-join maintenance and support join-map grouped aggregation, count-triggered refresh, and bounded semantic prompt execution
harden structured output with opt-in DeepSeek Responses JSON Schema predicates while preserving legacy model names and default paths
add benchmark accounting, supervision, and reproducible prompt-batching scale tooling

Validation

uv run ruff check src tests tools examples
uv run pyright
uv run python -m pytest -q -p no:cacheprovider (1714 passed, 20 skipped)
git diff --check origin/main...HEAD

Notes

all 14 commits include DCO sign-off
local unstaged and untracked research documents are not included
no new paid benchmark or provider run was performed for this PR

Summary by CodeRabbit

New Features
- Added exact count, sum, and avg aggregates with grouped queries, null handling, and incremental updates.
- Added limit, arithmetic expressions, conditional expressions, and safe casting.
- Added count-based refresh batching with pending-row inspection and manual flushing.
- Expanded semantic joins with key and predicate restrictions.
- Added LOTUS prompt batching, structured-output transports, and bounded JSON repair.
- Added the SemanticDataflow API and expanded public exports.
Documentation
- Updated design and evaluation documentation for aggregation, batching, refresh, and execution behavior.

- Promote TODO.md as the single current implementation and research backlog. - Rename the outdated next-step roadmap to a stable LOTUS lowering reference. - Remove superseded operator and differential-rule TODO documents. Signed-off-by: Von <haodong.lu.me@gmail.com>

- support general deterministic JoinOn restrictions for semantic joins - maintain append-only inner k=None semantic joins with binary state - preserve bag multiplicity and reject unsupported differential shapes - document and test the incremental semantic-join contract Signed-off-by: Von <haodong.lu.me@gmail.com>

- buffer normalized source rows until an explicit count boundary - apply each pending relation-valued delta atomically with flush support - preserve pending rows across failures and schema-v3 checkpoints - document and test eager compatibility and storage transactions Signed-off-by: Von <haodong.lu.me@gmail.com>

- add shared context-bounded prompt batching with strict task validation - support packed execution across semantic filter, map, flat-map, join, groupby, and aggregate paths - narrow semantic state reaggregation inputs and preserve explicit physical fingerprints - expose refresh and prompt-batch controls in benchmark CLIs with tests and API documentation Signed-off-by: Von <haodong.lu.me@gmail.com>

- add count, sum, and average aggregate descriptors - maintain exact hidden aggregate state across insertions and retractions - document the public API and cover direct and differential execution Signed-off-by: Von <haodong.lu.me@gmail.com>

- centralize the grouped-aggregate default - apply join-map consistently across Claude, Zep, and benchmark entry points - preserve explicit historical rules and the Mem0 compatibility path Signed-off-by: Von <haodong.lu.me@gmail.com>

Expose Source, Relation, and SemanticDataflow through the public API. Add try_cast and case_when expressions with serialization and evaluation. Provide defensive named-view reads and cover the public contracts. Signed-off-by: Von <haodong.lu.me@gmail.com>

…sport Centralize bounded JSON repairs and validate complete operator protocols. Add explicit DeepSeek Responses routing, schemas, usage and failure metadata. Version repair contracts for checkpoint isolation while retaining Chat defaults. Fix structured output reserves and reject oversized configured chunks instead of silently shrinking them. Cover transport, repair, batching, and benchmark configuration contracts. Signed-off-by: Von <haodong.lu.me@gmail.com>

Add fail-fast two-pass experiment scheduling and verified completed-condition reuse. Track cumulative physical usage budgets and immutable source provenance. Summarize and plot scale results while retaining mixed-version provenance. Expose trace-excluded insertion distributions and maximum latency with regression tests. Signed-off-by: Von <haodong.lu.me@gmail.com>

Stop owned process groups when monitoring or PID publication fails. Ensure status-write failures cannot bypass shutdown, preserving the original error. Cover failure cleanup and the existing SIGINT-to-SIGTERM escalation contract. Signed-off-by: Von <haodong.lu.me@gmail.com>

Share comparison operators across expression nodes instead of Python dataclass equality. Keep scalar expressions unhashable and preserve boolean literals and serialized query contracts. Verify composite comparisons and reflected operands against real DataFrame filtering. Signed-off-by: Von <haodong.lu.me@gmail.com>

Reject oversized local aggregate documents without empty prompts and recount after renumbering. Preserve ordered Responses failures when accounting metadata is malformed, retaining usable usage. Cover both aggregate paths and response fallbacks; correct two oversized legacy test fixtures. Signed-off-by: Von <haodong.lu.me@gmail.com>

Add an unordered relational limit with deterministic parent-state maintenance. Initialize global aggregate identity on empty parent updates. Allow provider normalization without assumed prices and cover the contracts with regression tests. Signed-off-by: Von <haodong.lu.me@gmail.com>

Add opt-in single-task JSON-schema predicates and preserve default execution paths. Accept current and legacy DeepSeek model names without rewriting requests. Wire explicit context capacity through execution provenance and support audited experiment continuations. Add offline predicate, configuration, and continuation regressions. Signed-off-by: Von <haodong.lu.me@gmail.com>

chatgpt-codex-connector · 2026-09-11T15:31:39Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-11T15:38:19.736098Z	`fda741c`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

coderabbitai · 2026-09-11T15:31:43Z

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: c897fac4-aba5-4ee3-82a3-336798f2dcea

📥 Commits

Reviewing files that changed from the base of the PR and between fda741c and 84dd59b.

📒 Files selected for processing (9)

src/agent_memory/adapters/lotus/deepseek_responses_lm.py
src/agent_memory/adapters/lotus/request_hook.py
src/agent_memory/adapters/lotus/structured.py
src/agent_memory/dataflow.py
src/agent_memory/runtime/executor.py
src/agent_memory/tracing/semantic.py
tests/test_dataflow_recovery_hook.py
tests/test_differential_runtime.py
tests/test_semantic_dataflow.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Walkthrough

This change adds numeric relational aggregates, scalar expressions, semantic join maintenance, count-based refresh, LOTUS prompt batching, structured-output repair, DeepSeek Responses support, benchmark wiring, and experiment tooling. Design documents and tests describe these capabilities.

Changes

Operator contracts and differential execution

Layer / File(s)	Summary
Public operators and planning `src/agent_memory/policy/`, `src/agent_memory/planner/`	Adds numeric aggregates, scalar expressions, `limit`, generalized `sem_join(on=...)`, algebraic aggregate state, and append-only semantic join maintenance.
Runtime and dataflow APIs `src/agent_memory/api.py`, `src/agent_memory/dataflow.py`, `src/agent_memory/runtime/*`	Adds `CountRefresh`, pending-row inspection, atomic `flush()`, `SemanticDataflow`, batch delta application, defensive copies, and schema-v3 checkpoints.

LOTUS execution

Layer / File(s)	Summary
Batching and structured output `src/agent_memory/adapters/lotus/prompt_batching.py`, `src/agent_memory/adapters/lotus/json_output.py`, `src/agent_memory/adapters/lotus/structured.py`	Adds task-keyed batching, context validation, retries, strict parsing, bounded JSON repair, validators, and repair metadata.
Semantic operators `src/agent_memory/adapters/lotus/sem_filter`, `src/agent_memory/adapters/lotus/sem_join.py`, `src/agent_memory/adapters/lotus/sem_groupby.py`, `src/agent_memory/adapters/lotus/sem_topk_join.py`, `src/agent_memory/adapters/lotus/sem_agg`	Adds batched filtering, joins, group matching, listwise joins, and semantic aggregation.
Native and relational execution `src/agent_memory/adapters/lotus/deepseek_responses_lm.py`, `src/agent_memory/adapters/lotus/context.py`, `src/agent_memory/adapters/lotus/relational.py`, `src/agent_memory/adapters/lotus/algebraic_aggregates.py`	Adds DeepSeek Responses JSON-schema execution, provider metadata, arithmetic and conditional expressions, `limit`, batched aggregates, and exact numeric aggregate execution.

Integration and evaluation

Layer / File(s)	Summary
Configuration and benchmark propagation `src/agent_memory/evaluation/`, `tools/evaluation/`	Propagates refresh, prompt batching, aggregate dispatch, structured transport, context limits, grouped-rule defaults, and driver `flush()` behavior.
Validation and experiment tooling `tests/*`, `tools/evaluation/prompt_batching_scale.py`, `tools/analysis/prompt_batching_scale_report.py`	Adds coverage for operators, runtime behavior, transports, batching, checkpoints, semantic joins, benchmark contracts, scale execution, and reports.
Design and dependency updates `README.md`, `docs/design/*`, `tools/evaluation/README.md`, `pyproject.toml`	Updates design documents, removes superseded TODO documents, documents current contracts, and adds `json5`.

Estimated code review effort: 5 (Critical) | ~90 minutes

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

_{Comment @coderabbitai help to get the list of available commands.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fda741ca90

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

- Reject views derived from undeclared sources, including matching schemas - Detach mutable object cells at ingestion and public read boundaries - Expose the existing checkpoint contract through SemanticDataflow - Keep differential runtime tests isolated from live providers Signed-off-by: Von <haodong.lu.me@gmail.com>

- Add deterministic request identities around DeepSeek Responses sends - Allow callers to persist or replay raw provider responses before returning - Avoid duplicate trace artifacts while an external provider journal is active - Cover request positions, transport guards, and checkpoint recovery offline Signed-off-by: Von <haodong.lu.me@gmail.com>

SleepyLGod added 14 commits September 1, 2026 22:21

Copilot AI lite review requested due to automatic review settings September 11, 2026 15:31

Copilot started reviewing on behalf of SleepyLGod September 11, 2026 15:32

View session

SleepyLGod self-assigned this Sep 11, 2026

SleepyLGod added bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request feature labels Sep 11, 2026

chatgpt-codex-connector Bot reviewed Sep 11, 2026

View reviewed changes

Comment thread src/agent_memory/dataflow.py

Comment thread src/agent_memory/runtime/executor.py Outdated

This comment was marked as duplicate.

Sign in to view

SleepyLGod added 2 commits September 12, 2026 01:02

SleepyLGod merged commit 2bdce24 into main Sep 11, 2026
5 checks passed

Navigation Menu

Repository navigation

[feat-semantic] Add differential semantic dataflow execution - #12

[feat-semantic] Add differential semantic dataflow execution#12
SleepyLGod merged 16 commits into
mainfrom
codex/semantic-prompt-efficiency

SleepyLGod commented Sep 11, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

chatgpt-codex-connector Bot commented Sep 11, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Sep 11, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as duplicate.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

SleepyLGod commented Sep 11, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Notes

Summary by CodeRabbit

Uh oh!

chatgpt-codex-connector Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

coderabbitai Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

This comment was marked as duplicate.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SleepyLGod commented Sep 11, 2026 •

edited by coderabbitai Bot

Loading

chatgpt-codex-connector Bot commented Sep 11, 2026 •

edited

Loading

coderabbitai Bot commented Sep 11, 2026 •

edited

Loading