Grasp Demo branch with sim - #3301
Draft
mustafab0 wants to merge 18 commits into
Draft
Conversation
A PD-held arm rests 1e-4..1e-3 rad off target, so microradian equality gates rejected every preview-then-execute pick and every plan requested while the arm merely held position (INVALID_START / 'no longer matches the stored plan start'). 0.01 rad absorbs hold jitter and still catches a genuinely moved arm.
… manip/grasp-sprint-july-26-sim # Conflicts: # dimos/manipulation/planning/world/roboplan_world.py # dimos/robot/manipulators/xarm/blueprints/simulation.py
Compose GraspGenXModule into xarm-grasp-sim so pick's provider path runs learned proposals (autoconnect fills the GraspGenSpec ref); CC's xArm gripper config from grasp_config. xarm-perception-sim is unchanged. Adds static_box_obstacles to ManipulationModuleConfig next to floor_z: startup box obstacles for scene furniture the planner must always respect. The grasp-sim table matches data/xarm_grasp_sim/scene.xml.
❌ 7 Tests Failed:
View the top 3 failed test(s) by shortest run time
View the full list of 1 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
Convert MuJoCo's internal closing-joint feedback to aperture values and keep direct gripper RPCs in the coordinator's hold-last cache. This prevents arm trajectory commands from closing a gripper that pick just opened. Increase the xArm simulation pre-grasp clearance from 5 cm to 10 cm and cover both regressions with focused tests.
pointcloud_to_convex_hull_obj centers the hull on the mean of the points, but the obstacle was placed at obj.pose, which carries the bounding-box centre. The two differ by ~1.6cm on tall asymmetric objects, so mesh obstacles sat offset from the geometry they represent.
Publish the object cloud, the ranked grasp candidates, and the candidate currently under feasibility check as Viser display layers, so a failing pick can be inspected instead of inferred from logs. The gripper geometry comes from config rather than the generator: grasp_gen_x bakes grasp_frame_to_tcp into candidate poses and the wireframe un-applies it, so the xarm blueprint sources both from one config object. Publishing is opt-in, and every call is suppressed so visualization cannot fail a pick.
Plumb the existing WorldObstacleMonitor use_mesh_obstacles option through start_obstacle_monitor and enable it for the grasp sim. Bounding boxes envelop an object at every height, so a side approach collides with empty space; the hull follows the observed geometry. Other blueprints keep boxes.
pointcloud_to_convex_hull_obj named its output hull_{id(points)}.obj. id() is
a memory address, and CPython reuses a freed address for the next same-sized
array, so sequential callers overwrote each other: five of six scene objects
collapsed onto one hull file and every obstacle took the shape of whichever
was written last. Intermittent, because it depends on allocator reuse.
The default name is now a uuid. The obstacle monitor passes an explicit path
per object instead, so rescanning overwrites in place rather than growing the
cache without bound.
The stored-plan freshness check rejected a grasp two milliseconds after IK solved it: the preview animation runs for ~1.6s between planning and executing, and a PD-held shoulder sags past the fixed 0.01 rad threshold in that window. That is not the stale-plan case the check guards against. The tolerance is now configurable (execution_start_tolerance, default 0.05 rad, 0 disables) and the rejection message reports the measured drift so the threshold can be set from data rather than guessed.
Pick suppressed every object obstacle for the whole transaction, so the approach was planned against an empty scene and swept through whatever stood between the arm and the target. Suppression is now staged. Candidate selection and the approach plan against the full scene; only the legs that must reach into and carry the target - grasp, close, retreat, and their feasibility check - hide the target, and nothing else is ever hidden. A failed restore is recorded on the transaction so it still surfaces, and pick_suppress_all_object_obstacles remains as a diagnostic escape hatch.
TomCC7
force-pushed
the
manip/grasp-sprint-july-26
branch
4 times, most recently
from
August 5, 2026 05:32
90216db to
c652e0c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A sim exploration of grasping