Skip to content

feat: run numeric WASM UDFs in SQL pipelines - #2524

Open
QIU-Guanzong wants to merge 1 commit into
getdozer:mainfrom
QIU-Guanzong:feat/wasm-udfs
Open

QIU-Guanzong wants to merge 1 commit into
getdozer:mainfrom
QIU-Guanzong:feat/wasm-udfs

Conversation

@QIU-Guanzong

@QIU-Guanzong QIU-Guanzong commented Sep 26, 2026 •

Copy link
Copy Markdown

Adds project-local AssemblyScript UDFs for #1653. The mapping from the issue now loads a .wasm module during pipeline setup, and SELECT my_function(field1, field2) calls its export through wasmi. The existing tagged UDF list remains supported.

/claim #1653

The initial ABI is fixed-arity i32, i64, f32, and f64, with one numeric result. Host conversions are checked, NULL inputs propagate, and each evaluation gets fresh guest state so updates and deletes reproduce their earlier values. Fuel, memory, table and module-size limits are bounded. The only allowed host import is AssemblyScript's env.abort; filesystem, network, clocks and WASI are unavailable.

The example includes pinned AssemblyScript source, a compiled module and readable WAT. Tests exercise the original YAML mapping through the actual SQL planner, schema propagation and DAG processor factories, including insert/update/delete, NULLs, a guest assertion and state isolation. Configuration errors and unsupported SQL call forms fail before streaming.

Validation on macOS with Rust 1.79.0 and protoc 29.3:

  • SQL library: 221 passed, one existing ignored test.
  • Expression library: 36 passed, including the WASM runtime tests.
  • UDF configuration tests: 4 passed.
  • Formatting and diff checks passed; Clippy completed with existing repository warnings.
  • Default-feature cargo check --locked -p dozer-cli and cargo build --locked -p dozer-cli passed. The actual CLI then started from the original WASM mapping, accepted three synthetic HTTP requests through its native localhost Webhook source, and logged 7, 8 and NULL at its native Dummy sink. All three HTTP responses were 200; SIGINT shut the process down with exit 0.

The library suites use --locked --no-default-features --lib; the separate CLI check/build and localhost run use defaults without trimming the CLI. External database sinks, checkpoint recovery and all optional features were not exercised. The original demo uses synthetic source events and a synchronous queue around real DAG processors; the additional CLI transcript records the actual asynchronous runtime and Webhook connector. Strings, objects, optional AssemblyScript parameters, --exportStart and multiple return values are outside this numeric ABI; relative paths follow Dozer's existing working-directory convention.

Demo video, actual transcripts, CLI configuration/inputs and hash manifests. The video is a paced rendering of the recorded test output; cli-transcript.txt and cli-verification.json provide the separate full CLI run evidence.

Developed and reviewed with assistance from OpenAI Codex. The review found and fixed computed-expression nullability and configuration/call-validation edge cases; the validation limits above are intentional disclosures.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants