feat(manipulation): benchmark RoboPlan OInK against Pink IK by Jerrybery · Pull Request #3334 · dimensionalOS/dimos

Jerrybery · 2026-08-03T04:19:01Z

Contribution path

Linked issue or discussion: Benchmark RoboPlan OInK against Pink for manipulation IK #3232

Stacked on #3230 (RoboPlan-native OInK backend) — base is cc/feat/roboplan-oink; retarget to main once #3230 lands.

Problem

RoboPlan worlds default to the bundled OInK backend while Pink remains the generic IK backend, but there was no comparable performance data for the two on DimOS manipulation workloads (latency, convergence reliability, accuracy, resource usage).

Solution

Add dimos/manipulation/benchmark_ik_backends.py, a reproducible benchmark, plus benchmark_ik_backends.md documenting methodology and observed tradeoffs.

One shared RoboPlanWorld per robot (xArm6, xArm7); Pink (world-agnostic) and OInK (the world itself) solve identical FK-sampled, collision-free, reachable targets with the same seed, tolerances, and max_attempts=10.
Wall time via time.perf_counter, success by IKStatus, and every successful solution independently re-verified through the world's FK with compute_pose_error.
200 timed solves per (robot, backend) after 10 warmups; results reproduced across two seeds.

Key findings (seed 0, 16-core x86_64):

robot	backend	success	p50 ms	p95 ms	mean ms	pos err p50 (mm)
xarm6	pink	87.5%	29.6	512.2	123.8	0.74
xarm6	roboplan_oink	84.5%	31.3	99.8	38.4	0.27
xarm7	pink	99.0%	8.6	207.5	38.9	0.74
xarm7	roboplan_oink	97.0%	5.1	66.5	16.4	0.53

OInK's mean latency is 2–3.5x lower and its p95 ~5x tighter; Pink's slow solves are its failures (restarts burning the iteration budget).
Pink succeeds slightly more often, and the two fail on almost disjoint target sets (6/200 shared on xArm6, 0 on xArm7) — an OInK-first, Pink-fallback chain would beat either backend alone.
Both honor the 1 mm tolerance on every reported success; per-solve RSS delta is negligible.

How to Test

uv sync --extra manipulation --inexact
uv run python dimos/manipulation/benchmark_ik_backends.py --samples 200 --warmup 10 --output /tmp/ik_bench.json

Requires the xarm_description LFS data. ruff format --check, ruff check, and mypy (strict) pass on the new files.

AI assistance

Kimi Code CLI (Moonshot AI) explored the IK stack, wrote the benchmark and documentation, and ran the benchmark on a remote dev machine under my direction. I reviewed the approach and results.

Checklist

I have read and approved the CLA.

TomCC7

Hi thx for the contribution. Benchmark looks decent to me. Added some comments on format etc. Also can you try tuning these ik backends a bit? For example in your result pink-ik has a better success rate while having a significantly longer p95 time (while p50 stays similar to oink), I think there's some correlation here and might be good to explore a bit.

TomCC7 · 2026-08-03T20:17:43Z

Need to move to a better place. maybe we should have a dedicated benchmark folder for these scripts... @paul-nechifor I have to spawn you here lol. Do you think we should put these benchmark scripts into pypi package so user can run them? Or we keep it outside and make a standalone benchmark folder?

Moved to a dedicated top-level benchmarks/ folder (benchmarks/ik_backends.py) in 6508673. Happy to relocate again depending on what you and @paul-nechifor decide re: shipping benchmarks in the pypi package.

TomCC7 · 2026-08-03T20:18:02Z

This file should not stay in the codebase. Or maybe we should convert it to a doc into doc folder that provides instruction on how to run the benchmarks, and maybe extensible to other ik backends in the future.

Removed from the codebase in 6508673 and converted into a docs page: docs/capabilities/manipulation/ik_benchmark.md. It covers how to run the benchmark and how to extend it (new scenarios, new IK backends) — no results baked in. Observed numbers and tradeoffs are posted as PR comments instead (see the tuning comment).

TomCC7 · 2026-08-03T20:24:06Z

+    return float(np.percentile(np.asarray(values), pct))
+
+
+def _summarize(run: BackendRun) -> dict[str, Any]:


better create a standard dataclass for result

Done in 6508673 — _summarize now returns a RunSummary dataclass, with a DistributionStats dataclass for the measured distributions (mean/p50/p95/max).

TomCC7 · 2026-08-03T20:44:05Z

+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)


can switch to typer which is our repo standard on cli parser: https://typer.tiangolo.com/

Done in 6508673 — CLI is now typer.

TomCC7 · 2026-08-03T21:01:18Z

+    for robot_name in args.robots:
+        print(f"[setup] building RoboPlanWorld for {robot_name} ...", flush=True)
+        config = ROBOT_CONFIG_FACTORIES[robot_name]()
+        world = create_world("roboplan")


We should create different world for evaluating different robots. There's no need to save resources here and robots in the same world might interfere with each other (for example collision checking).

Agreed — each robot already got its own world, and 6508673 goes further: every backend run now builds its own fresh RoboPlanWorld, so backends never share a mutable scene (no cross-backend interference via collision state or scratch contexts).

TomCC7 · 2026-08-03T21:09:10Z

+            "pink": create_kinematics(config=PinkKinematicsConfig()),
+            "roboplan_oink": create_kinematics(


better making these config also and make it easy to extend to other solvers

Done in 6508673 — solvers come from a SolverSpec registry (_solver_registry()); adding a backend is one entry (name + constructor from a fresh WorldSpec), selectable at runtime via repeatable --solver.

TomCC7 · 2026-08-03T21:10:32Z

+)
+from dimos.utils.transform_utils import pose_to_matrix
+
+ROBOT_CONFIG_FACTORIES = {


can you add dual arm test cases?

Added in fcaa22e — a dual_xarm6 scenario mirroring the dual_xarm6_planner blueprint (two xArm6, 1 m apart), solved as joint multi-target solve_pose_targets calls with scene-wide collision filtering during sampling. It exposed an interesting OInK convergence weakness on composite 12-DoF solves — details in the tuning comment.

…iew feedback - Relocate script to benchmarks/ik_backends.py (dedicated benchmark folder) - Replace results markdown with docs page on running/extending the benchmark - Switch CLI from argparse to typer (repo standard) - Return a RunSummary dataclass instead of a plain dict - Registry-based solver specs for easy extension to other IK backends - Build a fresh world per backend run so backends never share a mutable scene - Add --max-attempts/--pink-max-iterations knobs for tuning sweeps

greptile-apps · 2026-08-04T07:53:30Z

Greptile Summary

The PR adds a reproducible benchmark comparing Pink and RoboPlan OInK across single- and dual-arm scenarios.

Samples collision-free FK-derived targets and runs each solver in a fresh world.
Records latency, convergence status, independently verified pose error, and coarse RSS deltas.
Documents benchmark methodology, execution options, and extension points.
Adds the benchmark page to the manipulation documentation navigation.

Confidence Score: 4/5

The PR should not merge until the RSS fallback stops using a process-wide high-water mark that invalidates sequential backend comparisons outside Linux /proc environments.

The Linux /proc/self/statm path addresses the previously reported measurement defect, but the fallback still returns ru_maxrss; because backend runs remain sequential in one process, later runs can inherit an earlier backend’s peak and report understated or zero RSS growth.

Files Needing Attention: benchmarks/ik_backends.py

Important Files Changed

Filename	Overview
benchmarks/ik_backends.py	Adds scenario construction, target sampling, backend execution, independent FK verification, resource measurement, aggregation, and CLI output for the IK benchmark.
docs/capabilities/manipulation/ik_benchmark.md	Documents benchmark methodology, fairness controls, commands, requirements, and extension points.
docs/docs.json	Adds the IK benchmark page to the manipulation documentation navigation.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Build sampling world] --> B[Sample collision-free joint states]
    B --> C[Generate reachable FK targets]
    C --> D[Build fresh world per backend]
    D --> E[Run warmup solves]
    E --> F[Run timed solves]
    F --> G[Verify successful solutions with world FK]
    G --> H[Aggregate latency, success, accuracy, and RSS]
    H --> I[Print summary and optionally write JSON]

_{Reviews (3): Last reviewed commit: "fix(benchmarks): measure current RSS ins..." | Re-trigger Greptile}

greptile-apps · 2026-08-04T07:53:33Z

+        backend=solver.name,
+        records=records,
+        peak_rss_delta_mb=_peak_rss_mb() - rss_before,
+    )


Process-wide peak skews backend RSS

When multiple backend runs execute sequentially, ru_maxrss retains the process-lifetime high-water mark, so subtracting the snapshots before and after each run reports zero or understated memory growth for later backends and makes the per-backend resource comparison invalid.

Good catch — fixed in cc4e199. The benchmark now reads current RSS from /proc/self/statm before/after each backend run instead of the ru_maxrss high-water mark, so per-backend deltas are real. Re-ran xarm6 with the fixed metric: pink 1.1 MB, OInK 0.0 MB — the "per-solve memory footprint is negligible" conclusion is unchanged, now with a valid measurement.

- Scenario-centric design: one or more robots sharing a world - dual_xarm6 mirrors the dual_xarm6_planner blueprint (two xArm6, 1 m apart) - Multi-robot scenarios solved as joint multi-target solve_pose_targets calls - Scene-wide collision filtering during target sampling - Rename --robot to --scenario; docs updated

ru_maxrss is a process-lifetime high-water mark, so per-backend deltas were zero/understated for every backend after the first one peaked. Read /proc/self/statm for a real before/after delta (greptile P1).

Jerrybery · 2026-08-04T08:52:00Z

Tuning follow-up on the success-rate ↔ p95 correlation, plus a dual_xarm6 scenario and an attempts-cost analysis per use case. All numbers: 200 timed solves per config, 10 warmup, seed 0, shared FK-sampled reachable targets, check_collision=True, 16-core x86_64, pin-pink 4.2.0, roboplan 0.0.100.

1. Attempts sweep (Pink iters=200)

attempts	xarm6 pink	xarm6 OInK	xarm7 pink	xarm7 OInK
1	45.0% @ 32ms	25.5% @ 8ms	83.0% @ 18ms	53.0% @ 6ms
2	57.5% @ 54ms	41.5% @ 14ms	89.0% @ 24ms	67.5% @ 10ms
5	80.5% @ 88ms	63.0% @ 28ms	95.5% @ 31ms	91.0% @ 14ms
10	90.0% @ 125ms	84.5% @ 37ms	97.0% @ 30ms	97.0% @ 15ms

(success rate @ mean latency)

Your suspicion confirmed: both backends buy success with attempt budget. Pink wins per-attempt (80.5% vs 63.0% at attempts=5 on xarm6) but each attempt costs ~3x more, because a failed Pink attempt burns the full iteration budget while a failed OInK attempt costs ~2ms. At equal wall-clock budget OInK dominates (e.g. xarm7: OInK a2 = 67.5% @ 10ms mean vs Pink a1 = 83% @ 18ms — but OInK a5 = 91% @ 14ms beats both).

2. Pink `max_iterations` is mostly wasted budget (attempts=10)

pink iters	xarm6 success	xarm6 mean / p95	xarm7 success	xarm7 mean / p95
50	88.0%	46 / 153ms	97.0%	15 / 49ms
100	89.0%	64 / 253ms	97.0%	21 / 94ms
200 (default)	90.0%	125 / 458ms	97.0%	30 / 136ms

Cutting 200→50 keeps success within noise but cuts mean latency ~3x and p95 ~3x — the default 200-iteration budget is mostly burned on attempts that will never converge (safety_break already stops early when they do). iters≈50 looks like a strictly better Pink default for fallback use.

3. Dual-arm (`dual_xarm6`, joint multi-target, two xArm6 1 m apart)

backend	attempts	success	mean	p95	ms per success
pink	10	82.5%	228ms	552ms	276
OInK	10	36.5%	132ms	183ms	362
OInK	30	68.0%	272ms	516ms	400
OInK	100	91.0%	461ms	1697ms	507

OInK's composite 12-DoF solve usually burns all 100 iterations per attempt without converging (failed solves average 996/1000 iterations). More attempts do eventually converge (36.5→91%), so the targets aren't unreachable — the hardcoded per-attempt iteration cap (_MAX_ITERATIONS_PER_ATTEMPT = 100) looks too low for composite multi-robot solves. Exposing/raising it in the OInK backend is probably a better fix than piling on attempts (which destroys the tail: p99 1.7s at attempts=100).

4. Attempts cost by use case

ms per success = mean latency / success rate (expected cost per successful solve when re-rolling failures):

RL / batch sampling (throughput): single arm — OInK everywhere (xarm7: 87.5 vs 46.8 successes/s/core at a1; its ms/success is nearly flat across attempts, 11.4→15.8). Dual arm — Pink currently wins on throughput (3.6 vs 2.0–2.8 successes/s/core) because OInK needs attempts=100 to reach 91%.
Real-time / fast inference (latency bound): OInK attempts=1 has p99 ≈ 11ms and cheap failures — safe for control loops with a downstream fallback. Pink's p99 is ≥78ms at any attempts level and 300ms+ beyond a5 — not suitable for strict real-time. OInK dual-arm at attempts=100 hits p99 1.7s — unusable in a control loop, again pointing at the iteration cap.

Net: OInK attempts are a cheap success-rate knob (until composite multi-robot solves); Pink attempts are an expensive one. Suggested defaults per use case: RL batch single-arm → OInK a5–a10; fast single-arm sampling → OInK a1; strict real-time → OInK a1–a2 with fallback; dual-arm → Pink for now, or fix the OInK iteration cap.

feat(manipulation): add OInK vs Pink IK benchmark (dimensionalOS#3232)

e3f9e77

Jerrybery requested review from Dreamsorcerer, leshy, mustafab0, paul-nechifor and spomichter as code owners August 3, 2026 04:19

github-actions Bot added the first-time-contributor PR opened by an author who had not previously committed to this repository label Aug 3, 2026

TomCC7 self-assigned this Aug 3, 2026

TomCC7 requested changes Aug 3, 2026

View reviewed changes

TomCC7 assigned Jerrybery and unassigned TomCC7 Aug 3, 2026

greptile-apps Bot reviewed Aug 4, 2026

View reviewed changes

Jerrybery added 2 commits August 4, 2026 16:36

fix(benchmarks): measure current RSS instead of process-wide peak

cc4e199

ru_maxrss is a process-lifetime high-water mark, so per-backend deltas were zero/understated for every backend after the first one peaked. Read /proc/self/statm for a real before/after delta (greptile P1).

		return float(np.percentile(np.asarray(values), pct))


		def _summarize(run: BackendRun) -> dict[str, Any]:



		def main() -> None:
		parser = argparse.ArgumentParser(description=__doc__)

		"pink": create_kinematics(config=PinkKinematicsConfig()),
		"roboplan_oink": create_kinematics(

Uh oh!

Conversation

Jerrybery commented Aug 3, 2026

Contribution path

Problem

Solution

How to Test

AI assistance

Checklist

Uh oh!

TomCC7 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Uh oh!

greptile-apps Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jerrybery commented Aug 4, 2026

1. Attempts sweep (Pink iters=200)

2. Pink max_iterations is mostly wasted budget (attempts=10)

3. Dual-arm (dual_xarm6, joint multi-target, two xArm6 1 m apart)

4. Attempts cost by use case

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

greptile-apps Bot commented Aug 4, 2026 •

edited

Loading

2. Pink `max_iterations` is mostly wasted budget (attempts=10)

3. Dual-arm (`dual_xarm6`, joint multi-target, two xArm6 1 m apart)