feat: run numeric WASM UDFs in SQL pipelines - #2524
Open
QIU-Guanzong wants to merge 1 commit into
Open
QIU-Guanzong wants to merge 1 commit into
QIU-Guanzong wants to merge 1 commit into
Conversation
|
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds project-local AssemblyScript UDFs for #1653. The mapping from the issue now loads a
.wasmmodule during pipeline setup, andSELECT my_function(field1, field2)calls its export through wasmi. The existing tagged UDF list remains supported./claim #1653
The initial ABI is fixed-arity
i32,i64,f32, andf64, with one numeric result. Host conversions are checked, NULL inputs propagate, and each evaluation gets fresh guest state so updates and deletes reproduce their earlier values. Fuel, memory, table and module-size limits are bounded. The only allowed host import is AssemblyScript'senv.abort; filesystem, network, clocks and WASI are unavailable.The example includes pinned AssemblyScript source, a compiled module and readable WAT. Tests exercise the original YAML mapping through the actual SQL planner, schema propagation and DAG processor factories, including insert/update/delete, NULLs, a guest assertion and state isolation. Configuration errors and unsupported SQL call forms fail before streaming.
Validation on macOS with Rust 1.79.0 and protoc 29.3:
cargo check --locked -p dozer-cliandcargo build --locked -p dozer-clipassed. The actual CLI then started from the original WASM mapping, accepted three synthetic HTTP requests through its native localhost Webhook source, and logged7,8andNULLat its native Dummy sink. All three HTTP responses were 200; SIGINT shut the process down with exit 0.The library suites use
--locked --no-default-features --lib; the separate CLI check/build and localhost run use defaults without trimming the CLI. External database sinks, checkpoint recovery and all optional features were not exercised. The original demo uses synthetic source events and a synchronous queue around real DAG processors; the additional CLI transcript records the actual asynchronous runtime and Webhook connector. Strings, objects, optional AssemblyScript parameters,--exportStartand multiple return values are outside this numeric ABI; relative paths follow Dozer's existing working-directory convention.Demo video, actual transcripts, CLI configuration/inputs and hash manifests. The video is a paced rendering of the recorded test output;
cli-transcript.txtandcli-verification.jsonprovide the separate full CLI run evidence.Developed and reviewed with assistance from OpenAI Codex. The review found and fixed computed-expression nullability and configuration/call-validation edge cases; the validation limits above are intentional disclosures.