Video: https://youtu.be/4DKriauQ05g?si=INIHtNN7Frs6Tbzm
This repo is the simulator half of the lerobot_openarm pipeline. Recording, training and evaluation — real and sim — are run from lerobot_openarm (real: Steps 0-5, sim: Sim Steps 1-6). Here: the sim server, Isaac Lab Mimic, and the robot / scene assets.
cd ~/Stanley_ws/IsaacLab && conda activate env_isaaclabThe USD files (robot, chips can, plate-wiping props) are gitignored. They live on the Hugging Face Hub at
ethanCSL/openarm-isaaclab-assets,
using the same paths as in this repo, and are fetched automatically the first time an OpenArm robot or
task is loaded (public, no login needed). source/isaaclab_assets/isaaclab_assets/openarm_assets.json pins
the exact revision, so after a git pull any asset that changed is refreshed too. USDs you edited locally
are never overwritten; you get a warning instead. To fetch ahead of time (e.g. before going offline):
python scripts/tools/openarm_assets.py download # --force also discards local editsAfter changing the gripper or camera-mount scripts, rebuild their layers, then publish the new USDs and
commit the updated manifest so other machines pull them (needs hf auth login with a write token;
--dry-run lists the files first):
./isaaclab.sh -p scripts/tools/sysid/add_openarm_hand.py
./isaaclab.sh -p scripts/tools/sysid/add_openarm_camera_mount.py
python scripts/tools/openarm_assets.py upload -m "Rebuild hand layer"
git add source/isaaclab_assets/isaaclab_assets/openarm_assets.json && git commit -m "Update OpenArm assets"| Task | Use |
|---|---|
Isaac-PickUp-RedCube-OpenArm-IK-Abs-v1 |
Recommended. Pringles hand-over with the real look (real colours, room, camera noise, fitted contact physics) |
Isaac-PickUp-RedCube-OpenArm-IK-Abs-v0 |
Same task, plain sim look |
Isaac-PickUp-RedCube-OpenArm-IK-Abs-Mimic-v1 / -Mimic-v0 |
Mimic annotate / generate for the tasks above |
Isaac-PlateWipe-OpenArm-IK-Abs-v0 |
Plate wiping |
Isaac-InsertBattery-OpenArm-IK-Abs-v0 |
Battery insertion |
cd ~/Stanley_ws/IsaacLab && conda activate env_isaaclab
./isaaclab.sh -p scripts/tools/lerobot_sim_server.py --enable_cameras --headless \
--task Isaac-PickUp-RedCube-OpenArm-IK-Abs-v1 --task_mode handover \
--domain_randomization none \
--mimic_hdf5 logs/demos/pringles_sim_src_v00.hdf5- Start
lerobot-record/lerobot-rolloutonly after[lerobot_sim_server] ready on 127.0.0.1:5710. - Drop
--mimic_hdf5 ...for teleoperation and evaluation. - Restart the server after pulling changes.
Stop the sim server first (Ctrl-C).
cd ~/Stanley_ws/IsaacLab && conda activate env_isaaclab
# 2a. Annotate
./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/annotate_demos.py \
--task Isaac-PickUp-RedCube-OpenArm-IK-Abs-Mimic-v1 \
--task_mode handover --auto --from_states \
--enable_cameras --headless \
--input_file logs/demos/pringles_sim_src_v00.hdf5 \
--output_file logs/demos/pringles_sim_src_v00_annotated.hdf5
# 2b. Generate (domain randomization: visual)
./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
--task Isaac-PickUp-RedCube-OpenArm-IK-Abs-Mimic-v1 --task_mode handover \
--input_file logs/demos/pringles_sim_src_v00_annotated.hdf5 \
--output_file logs/demos/pringles_sim_mimic_v00.hdf5 \
--generation_num_trials 500 --num_envs 4 --enable_cameras --headless \
--enable_domain_randomization --domain_randomization_profile visualOther generate options:
# no randomization
./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
--task Isaac-PickUp-RedCube-OpenArm-IK-Abs-Mimic-v1 --task_mode handover \
--input_file logs/demos/pringles_sim_src_v00_annotated.hdf5 \
--output_file logs/demos/pringles_sim_mimic_v00.hdf5 \
--generation_num_trials 500 --num_envs 4 --enable_cameras --headless
# strong randomization: background swap + any colour + table position + can size
./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
--task Isaac-PickUp-RedCube-OpenArm-IK-Abs-Mimic-v1 --task_mode handover \
--input_file logs/demos/pringles_sim_src_v00_annotated.hdf5 \
--output_file logs/demos/pringles_sim_mimic_v00.hdf5 \
--generation_num_trials 500 --num_envs 4 --enable_cameras --headless \
--enable_domain_randomization --domain_randomization_profile full --randomize_object_sizeCheck the output for jolts no source demo has (a can lurch at the hand-over, ~15% of episodes), then run the
remove command it prints. It writes pringles_sim_mimic_v00_clean.hdf5:
./isaaclab.sh -p scripts/tools/check_mimic_demos.py --dataset_file logs/demos/pringles_sim_mimic_v00.hdf5Then convert to a LeRobot dataset with lerobot_openarm's mimic_to_lerobot.py (lerobot_openarm README, Sim Step 4).
cd ~/Stanley_ws/IsaacLab && conda activate env_isaaclab
# replay one demo (episode 0)
./isaaclab.sh -p scripts/tools/replay_demos.py \
--task Isaac-PickUp-RedCube-OpenArm-IK-Abs-v1 --task_mode handover \
--dataset_file logs/demos/pringles_sim_src_v00.hdf5 --select_episodes 0 --enable_cameras
# remove bad demos (here 3, 7 and 11-13) into a new file
./isaaclab.sh -p scripts/tools/remove_demos_hdf5.py \
--dataset_file logs/demos/pringles_sim_src_v00.hdf5 --episodes 3 7 11-13 \
--output logs/demos/pringles_sim_src_v00_clean.hdf5| Profile | Changes |
|---|---|
none |
nothing (recording source demos) |
visual |
lighting, object/table colour (hue kept), camera mount jitter, table height |
full |
visual + background swap (random skybox), any object colour, table position |
On -v1 / -Mimic-v1 the lighting change also applies to the photographed room and table.
- SAFETY HOLD in sim (
request is ... from the measured joint): add--robot.max_tracking_error=1.0tolerobot-record. - Robot self-collisions are off by default (the arms can pass through each other); turn them on with
OPENARM_SELF_COLLISIONS=1 ./isaaclab.sh -p .... - One sim server at a time on a 16 GB GPU.
- Disk: Mimic output is ~250 MB per generated demo (cameras included).
- New recording set: bump
_v00in the dataset / HDF5 names.
Gripper with the official finger offsets and foam pads (opens like the real one), real D435i bar + clamp ring + cable,
black mats, room backdrop fused from the real recordings, fitted robot colours and contact physics.
Tools and how each was measured: scripts/tools/sysid/README.md.
Legacy workflow (before lerobot_openarm; kept for reference, not maintained)
Record in Isaac Sim with Quest 3 Pro (dora + MuJoCo IK):
cd ~/Stanley_ws/dora-openarm-data-collection && source .venv/bin/activate
dora run dataflow-vr-mujoco-ros2.yaml --uvcd ~/Stanley_ws/IsaacLab && conda activate env_isaaclab
./isaaclab.sh -p scripts/tools/record_demos_openarm.py \
--task Isaac-PickUp-RedCube-OpenArm-IK-Abs-v0 --task_mode handover \
--dataset_file logs/demos/pickup_pringle.hdf5 \
--enable_cameras --num_demos 10 \
--teleop_device vr_joint_ros2_native --ros2_domain_id 1 --manual_saveAdd --real_arm to drive the real arms alongside (--real_arm_dataset <dir> also records the real cameras).
Convert an HDF5 to LeRobot (old converter):
cd ~/Stanley_ws/IsaacLab && conda activate env_isaaclab
python -u scripts/tools/convert_hdf5_to_lerobot.py \
--hdf5 logs/demos/pickup_pringle.hdf5 \
--output ~/Stanley_ws/IsaacLab/datasets/ethanCSL/openarm_visuomotor_VR_pringles_test \
--task "Pick up the Pringles can with the right arm, hand it to the left arm" \
--fps 30 --cameras right_wrist_cam wrist_cam body_camEvaluate a SmolVLA policy inside Isaac Lab (old eval loop):
conda activate lerobot && cd ~/Stanley_ws/IsaacLab
python scripts/imitation_learning/lerobot/smolvla_server.py \
--checkpoint ethanCSL/openarm_visuomotor_VR_pringles_V14_background_30hz \
--task "Pick up the Pringles can with the right arm, hand it to the left arm." --port 5556cd ~/Stanley_ws/IsaacLab && conda activate env_isaaclab
./isaaclab.sh -p scripts/imitation_learning/lerobot/eval_smolvla_jointspace.py \
--task Isaac-PickUp-RedCube-OpenArm-IK-Abs-v0 --task_mode handover \
--num_rollouts 5 --horizon 300 --enable_cameras --cameras right_wrist_cam,wrist_cam,body_cam