dimos bake: compose rust native modules into one host binary - #3333
Open
leshy wants to merge 4 commits into
Open
dimos bake: compose rust native modules into one host binary#3333leshy wants to merge 4 commits into
leshy wants to merge 4 commits into
Conversation
CLI overrides like --robot-ips never reached worker processes, so modules deployed to workers opened zenoh sessions with no connect endpoints and silently failed to dial the robot on multicast-filtered LANs. Same fix as 1ec9a6ec2 on ivan/feat/body_obstacle.
…itting Ported from ivan/feat/body_obstacle (1ec9a6ec2): native rust processes get DIMOS_ZENOH_CONNECT with the same endpoints the python sessions dial, so the raycaster/MLS no longer depend on multicast scouting the LAN filters. Also the voxel_ray_tracer emission fix -- repeatedly-hit thin clusters (chair legs, box edges) stayed live-skipped below support_min and vanished from the cloud after their first frame.
Groundwork for `dimos bake`, which links several native modules into one binary. Three pieces, none of them useful alone: Module structs move out of main.rs into their crate's lib (module.rs) so a host can link them; main.rs is now a shim. pyo3/numpy go behind a `python` feature (default on) so a host can drop them and keep libpython symbols out of a static musl build. Each crate declares its ports in [package.metadata.dimos.module.<id>]. bake reads that as pure TOML, so it can draw and check the graph before compiling anything. #[module(name = "<id>")] reads the same table at expansion time and fails the build when the struct's #[input]/#[output] fields drift from it. dimos_module::host runs N modules in one process: one shared transport (type-erased through SharedTransport, since Transport returns impl Future and each module is monomorphized separately), one thread and runtime per module, configs all parsed and validated before anything spawns, and fail-fast when any module stops. Topic suppression rides the existing QoS channel as zenoh Locality::SessionLocal; LCM says so and ignores it. run_fallible now shares its build/subscribe/teardown path with the host via run_module_core.
`dimos bake ray-tracing mls-planner -o dist/go2-nav` reads the Cargo.toml registry, wires the modules the way autoconnect would, prints the graph, generates a crate under target/dimos-bake/ and builds it. The graph is drawn before anything compiles, so a bad composition costs a second, not a build. Two deliberate differences from python autoconnect: a same-name/different-type pair is an error (with the --remap that fixes it) rather than a silent disconnect, and --suppress refuses a topic the host does not publish. Topics carry the message type, matching the zenoh key a python NativeModule actually hands its native process — without that a standalone host sits on keys nobody else uses. baked_host() gives the blueprint side one NativeModule whose ports are the union of its members', so autoconnect, .remappings() and .namespace() are unchanged; only the stdin blob nests, one section per member. NativeModule grew a _stdin_blob() seam for it. The zenoh runtime refuses a current_thread scheduler, so every module runtime is multi-thread — `threads` sets its worker count rather than choosing a flavour.
leshy
requested review from
Dreamsorcerer,
mustafab0,
paul-nechifor and
spomichter
as code owners
August 2, 2026 23:00
❌ 4 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One robot-side process per rust module means one zenoh listen port per module, annoying deploy mechanics, py deps
I present dimos bake
prints the derived wiring before building the autoconnect graph, computed from Cargo.toml metadata alone:
registry: crates self-declare ports via
[package.metadata.dimos.module.<id>];#[module(name = "...")]fails compilation if metadata drifts from the actual#[input]/#[output]fieldswiring: identical to python autoconnect but in-process. internal hops stay on zenoh (same-session delivery) so every intermediate topic is externally observable unless suppressed (
Locality::SessionLocal), baked default overridable at runtimescheduling: one tokio runtime per module (
threads\metadata knob), fail-fast supervision - first module death takes the host downcontract: the baked host speaks the NativeModule protocol (stdin JSON, `DIMOS_*` envs), nested per-module sections;
baked_host()gives the python coordinator a drop-in module class with the union of member ports;--emit-configwrites a fully-defaulted config for systemd deploymentsFirst two commits are prerequisites from field-debugging the go2 zenoh link: worker processes now inherit the full host GlobalConfig (CLI
--robot-ipsnever reached workers), and native processes get explicitDIMOS_ZENOH_CONNECTendpoints instead of relying on multicast scouting that LANs filter. The thin-obstacle raycaster fix rides in the same commit (repeatedly-seen thin clusters vanished from the emitted cloud after their first frame).Solo per-module binaries are unchanged - bake is a deployment choice, not an architecture commitment.
Known wart (pre-existing):
mls_planner.node_edgesisnav_msgs.Pathin rust vsLineSegments3Din the python wrapper; python wins when it drives (topics come over stdin), standalone hosts use the rust spelling.