feat(sql): support incremental IN subqueries

[QIU-Guanzong]

/claim #1659

This adds the remaining dynamic IN (SELECT ...) path. For example, an outer row that arrives before its matching inner value is emitted when that value is inserted, then retracted when its last inner match is deleted. Outer updates, duplicate values, NULLs and empty inner results follow the same streaming path.

The planner connects a two-input membership stage to the existing WHERE processor. It retains outer-row multiplicity, counts inner values, and removes the internal nullable membership column before projection. The implementation covers IN/NOT IN within compound predicates, scoped CTEs, multiple/nested subqueries, deterministic single-column expressions, UNION, grouped aggregates and joins.

The tests exercise SQL parsing, DAG schema propagation and the real processor factories using synthetic source events. They also cover three-valued Boolean logic, qualified-name binding, exact row representation, both shared-source fanout orders, and explicit rejection of query shapes the existing engine cannot replay correctly.

Validation

• SQL library: 262 passed, 1 existing ignored; expression library: 34 passed. This includes 30 SQL-to-processor integration tests and a 1,200-event snapshot oracle for the membership processor.
• cargo fmt --all -- --check and git diff --check pass; the six-event demo also passes on the submitted commit.
• Strict scoped Clippy stops on existing dozer-types warnings. With --no-deps, two existing lints remain (unnecessary_to_owned and legacy_numeric_constants); allowing only those two categories passes. Full workspace/all-feature CI has not been run locally. The additional native localhost connector check below uses the default CLI.

cargo +1.79.0 test --locked -p dozer-sql -p dozer-sql-expression --no-default-features --lib

The local run used macOS and an existing protoc 29.3 installation. Cargo.lock and dependencies are unchanged.

Native CLI check

The default-feature CLI build also passes on the same submitted commit. Its real Webhook connector and asynchronous runtime processed 26 synthetic localhost requests across separate IN and NOT IN runs. A snapshot oracle matched every cumulative insertion stream logged by the native Dummy sink, including duplicate inner values, NULL/empty cases, batches, and deletion followed by reinsertion. Both processes shut down with exit 0 and closed their listeners.

The Dummy sink logs inserts only, so this additional run does not directly observe output retractions or claim recovery/performance coverage. The library tests above verify the full emitted operation stream. Reproduction script, configuration, transcript and verification record.

Demo

Watch the six-event demo · Transcript and SHA-256 manifest

The video is a paced rendering of actual output from the SQL-to-DAG demo below. It uses synthetic events and a synchronous test queue; it does not claim validation of live connectors, the full asynchronous executor, or restart recovery.

cargo +1.79.0 test --locked -p dozer-sql --no-default-features --lib \
  tests::in_subquery::membership_demo_with_synthetic_rows -- --exact --nocapture

Limits

Correlated and multi-column subqueries are rejected. SELECT DISTINCT, global aggregates without GROUP BY, unsupported query modifiers, volatile expressions on membership-dependent paths, and TTL rows without explicit deletions are also rejected. Membership-dependent UNION DISTINCT rejects the opt-in probabilistic set mode because it cannot reliably reproduce retracted row representations. Internal field-name collisions produce an error. Inner membership changes currently scan the retained outer rows; no performance or recovery guarantee is claimed.

AI assistance: implementation, tests and source reviews were prepared with OpenAI Codex; Claude Code also reviewed the additional CLI verification harness.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[QIU-Guanzong]

Sure