Skip to content

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

MuJoCo Robotic Simulation

Pick-and-place on a Franka Emika Panda in MuJoCo, end to end: a randomised task, a scripted expert that solves it, a LeRobot dataset recorded from that expert, an ACT policy trained on that dataset, and an evaluation that scores the policy against the expert's ceiling on tasks it never saw. Alongside it sit an earlier SAC + HER curriculum track and webcam hand teleoperation that emits the same 8-D action everything else does.

the scripted expert solving one episode

The two tracks

Both build on the same environment, panda_env.py, which depends only on mujoco, gymnasium and numpy — no learning framework. That separation is what lets the LeRobot environment, the data-generation workers and the teleop loop import an env without dragging in stable-baselines3 or firing a CUDA probe on import.

Imitation learning (the current line). Scripted expert → LeRobot dataset → ACT → evaluation against the expert. Every stage is measured with the same success criterion, so the numbers compare.

SAC + HER + curriculum (the earlier line). train_robot_HER_curriculum.py, learning from scratch with hindsight relabelling and a three-stage curriculum. Still runnable; see the note on the checked-in policy.

File Role
panda_env.py The environment: task distribution, control law, observations, reward
scripted_expert.py Phase-based PI/IK controller — validates the task, sets the ceiling, records demos
build_dataset.py Rolls the expert out into a LeRobotDataset (video + state + action)
train_act.py Launches LeRobot ACT training, with a Windows checkpoint fix
eval_policy.py Scores a trained policy on held-out seeds; --sweep gives success vs. steps
render_episode.py Renders one episode to a captioned GIF — expert or learned policy
train_robot_HER_curriculum.py SAC + HER + curriculum training and test rollout
view_panda.py Opens the scene in the MuJoCo viewer, no policy
teleop/ Webcam hand teleoperation — see Teleoperation
tests/ pytest suite for the IK collision constraints and the depth geometry
franka_emika_panda/ Panda MJCF; scene.xml is what everything loads

Environments

The pipeline spans two conda environments, because LeRobot and stable-baselines3 want different torch builds. Both carry mujoco and gymnasium, so panda_env.py imports in either.

ml_env lerobot_env
Interpreter ...\anaconda3\envs\ml_env\python.exe ...\anaconda3\envs\lerobot_env\python.exe
Python / torch 3.11.16 / 2.14.0+cu130 3.11.16 / 2.10.0+cu128
Also has stable-baselines3 2.9, mediapipe, mink, Pillow lerobot 0.4.4
Runs train_robot_HER_curriculum.py, scripted_expert.py, render_episode.py, python -m teleop, pytest build_dataset.py, train_act.py, eval_policy.py

Do not use the conda base environment. Its torch 2.5.1+cu121 has no kernels for sm_120: torch.cuda.is_available() returns True and then the first real tensor op fails. See GPU setup.

For a fresh machine:

python -m venv .venv
.venv\Scripts\activate           # Windows
# source .venv/bin/activate      # macOS/Linux

pip install -r requirements.txt
# then a torch build matching your GPU -- see GPU setup below

requirements.txt covers the simulation, RL and teleop stack. Two things it deliberately leaves out: torch, because the right build is machine-specific, and lerobot, because the imitation-learning stages live in their own environment (pip install lerobot). render_episode.py also needs Pillow, which arrives with stable-baselines3[extra].

The task

A red block sits somewhere on the table; a green marker shows where it should end up. Move it there.

Both the block pose and the goal are re-sampled every reset, so a policy faces a distribution of tasks rather than one instance. This was not always true: the env used to restore a single block spawn and never move the goal, which left exactly one goal to learn and nothing for HER's relabelling to do.

Block spawn uniform over x ∈ [0.35, 0.65], y ∈ [−0.25, 0.25], resting on the table (z = 0.225)
Goal same xy region; half on the table, half floating 5–20 cm above it
Separation ≥ 10 cm between block and goal, by rejection sampling, so no episode starts solved
Success ‖block − goal‖ < 0.05 m

Half the goals float on purpose, following the FetchPickAndPlace protocol: a table-only distribution lets a policy collapse onto sliding, an air-only one onto lifting. Keeping both in one distribution forbids either shortcut.

The xy region is smaller than the table (which spans x ∈ [0.25, 0.75], y ∈ [−0.35, 0.35]) so the arm never has to work at the edge of its reach. It was verified reachable: damped-least-squares IK places the TCP within 0.1 mm of every point on a 5×5 grid over it, collision-free.

Control rate and episode budget

One env.step holds data.ctrl constant for FRAME_SKIP = 20 physics steps of 2 ms each — a zero-order hold at 25 Hz, which is where manipulation policies in the literature live. Actions are 7 joint-velocity commands scaled by the Franka datasheet limits, plus a gripper command:

target_qpos = qpos[:7] + action[:7] * JOINT_VEL_LIMIT * control_dt

Referencing measured qpos rather than the previous target means a blocked arm cannot wind up an ever-growing position error.

MAX_EPISODE_STEPS = 200 is 8 s of robot time, sized from measurement rather than taste: the scripted expert needs a median of 81 steps and a p90 of 129 to finish, so 100 would have truncated most successful trajectories.

Both numbers replaced worse ones. The old env stepped physics once per action at 500 Hz for 1500 steps — 1500 near-identical decisions covering only 3 s of motion, i.e. less robot time spread over 7.5× more decisions, with a brutal credit-assignment horizon bought for nothing. It also commanded up to 50 rad/s, roughly 23× the real joint limit, so the position servo simply saturated against forcerange and the action had no physically meaningful scale.

Observations

Two observation sets, because two kinds of policy consume them.

State (observation, 23-D) — joint positions and velocities, TCP position, block position, gripper z-axis, plus achieved_goal / desired_goal in Gymnasium's GoalEnv shape, which is what HER relabels.

Vision — enabled by passing camera_names to the constructor. Adds pixels/external, pixels/wrist and a 19-D proprio vector: joint positions, joint velocities, finger apertures, and the TCP position from forward kinematics. That is only what a real robot reports about itself. The block position is deliberately absent — it is read straight out of the simulator, no sensor produces it, and handing it to a vision policy leaks the answer and produces a number that will not survive contact with a real robot. The policy has to find the block in the pixels.

Rendering is off by default (~0.6 ms per frame per camera) and the renderer is built lazily, so constructing an env never opens an OpenGL context it will not use — SubprocVecEnv workers would otherwise each open one.

The tcp site in panda.xml is the grasp point. Averaging the two finger body origins, as the env used to, lands ~45 mm behind the fingertip pads, so the old "gripper position" was never the point that touches anything.

The imitation-learning pipeline

1. Scripted expert

python scripted_expert.py eval  --episodes 300     # score it
python scripted_expert.py demos --episodes 500     # -> panda_demos.npz (flat arrays)

A six-phase controller — approach, descend, grasp, lift, transport, place — driven by PI resolved-rate IK on the TCP site. It exists for three reasons:

  1. Validate the environment. Prove the task is solvable within the episode budget before spending GPU-hours on it. A flat RL learning curve on an unsolvable task tells you nothing about your algorithm.
  2. Provide a ceiling. "SAC reached 60%" means nothing until you know whether the achievable number is 65% or 99%.
  3. Generate demonstrations. Its (observation, action) pairs are exactly what behaviour cloning, ACT or Diffusion Policy train on.

It scores 99.3% [97.6, 99.8] on 300 held-out tasks (Wilson interval — the normal approximation is wrong this close to 1).

Two implementation details cost the most debugging:

  • The outer loop must be PI, not P. The Panda actuators are pure PD position servos with no gravity compensation, so holding a pose against gravity needs a standing ctrl − qpos offset. A proportional controller drives its command to zero as error shrinks, so the arm stalls 1–3 cm short, worst at the far edge of the workspace where it is most extended. The integral term took success from 25% to 88%.
  • The task is 5-DOF, not 6. Position plus the two rotations that tip the gripper axis downward. Yaw is excluded deliberately: adding it as a zero-target row locks the wrist and wastes a DOF the reach needs. An earlier version pinned yaw to the wrong value and silently commanded a 90° rotation that fought the position servo all episode — 0% success.

Phases 4 and 5 are split for a related reason: driving the TCP straight down onto a table goal at full speed pushes the block into the surface until it pops out of the fingers, which accounted for every remaining failure. Approaching from above and descending at reduced speed took table goals 86.8% → 100%.

2. Dataset

# in lerobot_env
python build_dataset.py --episodes 500 --out data/panda_pick_v1

Writes a LeRobotDataset directly — no intermediate .npz bridged between two conda environments.

Feature Shape What it is
observation.images.external video, 96×96×3 third-person RGB, scene context
observation.images.wrist video, 96×96×3 wrist RGB, what the fingers are doing
observation.state 19 proprioception only (see above)
observation.environment_state 3 goal position
action 8 7 joint-velocity commands + gripper

The goal goes in observation.environment_state (LeRobot's FeatureType.ENV, which ACT consumes via env_state_feature) rather than being concatenated onto the robot state, because it is a property of the task, not of the robot.

Only successful episodes are recorded — a failed demonstration teaches the wrong thing. Demo seeds start at 10000, well above the evaluation range, so training and evaluation tasks can never overlap. The dataset currently in data/ is 490 episodes / 43,794 frames from 500 attempts, at 25 fps.

Two Windows-specific choices are baked in: parallel_encoding=False (LeRobot's default spawns a ProcessPoolExecutor per episode which, combined with SVT-AV1, intermittently dies with BrokenProcessPool — a 3-episode smoke test passed, a 500-episode run died on the third) and --vcodec h264 (lighter, and pyav — the only decoder available here — handles it fastest; LeRobot validates the name against its own list, so libx264 is rejected).

3. Train ACT

# in lerobot_env
python train_act.py                            # defaults below
python train_act.py --steps=40000 --batch_size=16
python train_act.py --policy.chunk_size=100    # override any default

A thin wrapper around LeRobot's own CLI: everything is passed through, and the defaults are applied only for flags you do not supply.

Default Value Why
--policy.chunk_size / --policy.n_action_steps 50 2 s of committed motion at 25 Hz. LeRobot's default of 100 is 4 s — more than half a median episode, which throws away most of the closed-loop correction chunking exists for
--save_freq 2000 A run killed at step 750 left nothing evaluable with the 5000 default. Checkpoints are ~350 MB; saving 3× more often is cheap insurance
--num_workers 12 Measured: data_s 0.722 → 0.104 → 0.038 at 4/8/12 workers, taking the loop from dataloader-bound to GPU-bound (updt_s 0.143). 1.64 → 2.97 step/s
--steps 60000
--batch_size 16
--output_dir outputs/act_v1

The Windows fix. LeRobot writes its checkpoints/last pointer with Path.symlink_to(), which raises OSError: [WinError 1314] A required privilege is not held by the client unless the process is elevated or Developer Mode is on. It fires inside the checkpoint-saving path, so every save crashes training: a 300-step probe completed all 300 steps and then died writing the pointer. train_act.py patches the reference the training script holds and falls back to a directory junction (mklink /J), which needs no privilege. Nothing in the installed LeRobot package is modified.

4. Evaluate

# in lerobot_env
python eval_policy.py outputs/act_v1/checkpoints/last/pretrained_model
python eval_policy.py outputs/act_v1 --sweep --episodes 100

Rolls the policy out in the environment the demonstrations came from and reports success with a Wilson interval, printed next to the expert's ceiling.

Two things make that comparison honest, and both are easy to get wrong:

  1. Identical success criterion. ‖block − goal‖ < 0.05, exactly what the expert uses. The env also carries _get_info's curriculum flags, but those belong to the SAC/HER track — stage 1 is gripper_to_block < 0.2. Scoring the policy with one criterion and the expert with another produces two numbers that cannot be compared.
  2. Disjoint seeds. Evaluation uses seeds 0…N−1; demonstrations start at 10000. No evaluated task was ever trained on.

--sweep scores every checkpoint in a run, which is the curve that matters: loss keeps falling while a policy that never closes the gripper in time stays at 0%.

One inference detail is worth knowing, because getting it wrong scores a hard 0% no matter how well the network fits. ACTPolicy does not normalise its own inputs or outputs in LeRobot 0.4.4 — normalize_inputs / unnormalize_outputs moved out of the model into separate processor pipelines saved beside the weights. eval_policy.py builds them with make_pre_post_processors (not PolicyProcessorPipeline.from_pretrained, whose postprocessor operates on a PolicyAction tensor and raises EnvTransition must be a dictionary. Got Tensor), and runs obs = preprocessor(obs); a = policy.select_action(obs); a = postprocessor(a).

5. Watch it

python render_episode.py --seed 19                       # scripted expert
python render_episode.py --seed 19 --policy outputs/act_v1/checkpoints/last/pretrained_model
python render_episode.py --seed 19 --width 640 --height 480 --stride 2

Offscreen-renders one frame per control step, captions each with the active phase and the block's lift height, and writes a looping GIF. A success rate tells you a policy fails 30% of the time; a GIF tells you it fails because it closes the gripper 2 cm early. Rendering the failing seeds is usually the more informative choice.

SAC + HER + curriculum

python train_robot_HER_curriculum.py train        # parallel envs, headless
python train_robot_HER_curriculum.py train --resume
python train_robot_HER_curriculum.py train-viz    # single env, viewer open
python train_robot_HER_curriculum.py test         # roll out the saved policy
tensorboard --logdir panda_her_tensorboard
Flag Meaning
--timesteps N Steps to run this session (default 250000)
--envs N Parallel envs (default: CPU cores − 1, capped at 16)
--resume Continue from panda_her_curriculum_final.zip + saved curriculum stage
--save-buffer Persist the HER replay buffer across runs (correct, but a multi-GB file)
--episodes N Episodes for the test rollout
--no-test Skip the test rollout after training

Difficulty advances automatically once the success rate over the last 100 episodes exceeds 80%:

Stage Goal Success criterion
1 Move the gripper near the block gripper within 20 cm of block
2 Get near the block with the gripper pointed down within 10 cm and orientation ≥ 0.90
3 Move the block to the target block within 10 cm of goal and orientation ≥ 0.90

The reward is a weighted sum of approach, placement and orientation terms, reweighted per stage in compute_reward.

Resuming

--resume restores three things that must stay in sync: policy weights from the .zip, the curriculum stage from panda_her_training_state.json (without which every resume silently restarts at stage 1), and the step counter, so TensorBoard curves stay continuous. Training also checkpoints on Ctrl-C, so an interrupted long run is resumable.

The replay buffer is not in the saved .zip. By default a resumed run collects RESUME_WARMUP_STEPS fresh transitions before restarting gradient updates, since otherwise SAC would train against an almost-empty buffer. --save-buffer carries real experience across runs instead, at the cost of a large file on disk.

safe_learning_starts() raises learning_starts when needed: HER cannot sample until at least one episode has finished, and SB3 counts timesteps in units of n_envs, so the real minimum is MAX_EPISODE_STEPS * n_envs.

The checked-in SAC policy is stale

panda_her_curriculum_final.zip is tracked so a fresh clone can run test mode — but it was trained before the environment was rewritten. It comes from the 1500-step, 500 Hz, qpos + a * dt * 50 control law with a fixed block spawn and a stationary goal. The network shapes still match, so SAC.load succeeds and the rollout runs, but the behaviour was learned under different dynamics on a different task. Treat it as an artifact, not a baseline; retrain before quoting a number from it.

The stage-1 note also still holds: HER is inert until stage 3. The stage-1 reward uses only the approach term and stage-1 success is gripper_to_block < 0.2; neither reads desired_goal, so goal relabelling changes nothing until the placement term is active.

Teleoperation

Drive the arm with your hand in front of a webcam — no leader arm, no VR rig. The teleop/ package tracks one hand, retargets it to a gripper target, solves IK, and emits the same 8-D action the policies use, so a session here could become demonstration data without a conversion step.

Setup

pip install mediapipe mink

mkdir -p teleop/models
curl -L -o teleop/models/hand_landmarker.task \
  https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/latest/hand_landmarker.task

The 7.8 MB model file is gitignored; HandTracker prints this command if it is missing.

Running

python -m teleop cameras     # which capture index is your webcam today
python -m teleop preview     # tracking only, no robot  <- start here
python -m teleop teleop      # drive the arm

preview exists as its own mode deliberately: tracking quality in your lighting is the assumption everything else rests on, and it costs nothing to check before a robot is involved.

Key Action
SPACE engage / release the clutch
R recenter — re-anchor your hand without moving the gripper
[ ] decrease / increase mapping scale
D toggle the depth axis on / off
, . decrease / increase the depth gain
I / O / P invert the horizontal / vertical / depth axis
Q or Esc quit

How it works

camera ─→ MediaPipe Hands ─→ One-Euro ─→ clutched relative map ─→ mink IK ─→ env.step()
                  │
                  └─ image landmarks ÷ world landmarks ─→ depth (optional)
  • One model, not two. Hand landmark 0 is the wrist and landmarks 4/8 are the fingertips, so a single HandLandmarker yields both the position signal and the gripper signal.
  • One-Euro filter, not a plain EMA. It adapts its cutoff to hand speed: heavy smoothing at rest kills the jitter that would make the arm buzz, light smoothing in motion keeps the lag you would otherwise feel while reaching.
  • Relative mapping with a clutch. The target is the TCP at engage plus a scaled offset from your hand at engage — so there is no jump on engage, a dropped frame parks the arm instead of throwing it, and re-clutching extends range the way lifting a mouse does.
  • mink, not hand-rolled DLS. mink solves a QP per step and enforces joint position and velocity limits inside the solve. Plain damped least squares handles singularities but is blind to joint limits, and a human waving at a camera drives the arm to its limits constantly.
  • The table and the arm's own links are constraints, not hopes. The same QP carries collision rows, so the solver will not command a step that drives the gripper through the work surface or folds the wrist back into the shoulder. This is not only about abuse: reaching a perfectly legal target at (0.5, 0.15, 0.30) transiently dips the fingertips 3.7 mm into the table without it, because differential IK follows the straightest joint-space path and not a straight line in the workspace. Tracking accuracy and settling time are unchanged. --no-avoid-collisions turns it off.
  • Plane by default, depth by choice. A single RGB camera measures its two image axes well and depth badly, so the default drives a plane (--plane frontal or table) with the third axis pinned and the gripper held pointing at the table. --depth frees that third axis; D toggles it mid-session without moving the gripper.
  • Depth from the ratio of two landmark sets, not from apparent size. See below — this is the part most likely to be got wrong.

Because ArmIK's velocity limit is exactly the env's JOINT_VEL_LIMIT, the action derived from the IK solution saturates at exactly 1.0 and is never silently clipped.

The depth axis

MediaPipe returns two sets of landmarks from one inference: image landmarks in pixels, and world landmarks in metres with the hand's translation removed. The image set is shape × orientation × distance tangled together; the world set is shape × orientation alone. Dividing one by the other cancels shape and orientation and leaves pixels-per-metre, which is a pure function of distance:

distance  =  focal x hand size x foreshortening ÷ palm span in pixels

where the last two come from the world landmarks and the first from the image landmarks. Foreshortening is what the world landmarks are really for, and it is worth separating from the hand's size — see Two divisions, not one below.

The division is the whole point. Apparent palm width alone is not a depth measurement — it foreshortens as cos θ when the wrist rolls, and a natural 25° of pronation while reaching forward reads as 10% farther away. Measured against a synthetic hand at a known 0.60 m (tests/test_teleop_depth.py):

wrist roll palm width alone this method
10° +9 mm +0.2 mm
25° +61 mm +0.6 mm
40° +179 mm +1.0 mm
50° +324 mm +1.4 mm

What keeps the residual that small is that every pair among the five palm landmarks contributes, so the near and far halves of a tilted hand average against each other and the weak-perspective error mostly cancels. Pitching the fingers toward the camera is the worse case, peaking near 1.8% (~11 mm) around 40°, because pitch tips out of the image plane the axis the palm is longest along.

Two divisions, not one. The naive version of this divides the image span by the world landmarks' projected span and stops there. That is wrong in a way that only shows up as jitter. The projected span is really two things multiplied — the hand's true size, which is constant, and the foreshortening, which is what we want — and absolute size from a single image is precisely the ambiguous half of the problem, so the network re-guesses it every frame and that guess is noisy. Dividing by it hands the noise straight to the arm.

The same landmarks' full 3D span isolates that guess: pairwise 3D distances are rotation-invariant, so it is size alone. Dividing the projected span by it leaves pure foreshortening and cancels the scale noise, and the hand's size then comes from a long running median — legitimate exactly because a hand's size does not change. Two consequences, both measured:

world-landmark size noise one division two divisions
2% 4.6 mm 0.9 mm
5% 10.4 mm 1.4 mm
10% 20.3 mm 2.5 mm
15% 30.5 mm 2.9 mm

At the default gain, 10% size noise is 0.9 mm of gripper jitter rather than 7.1 mm. The size median runs over 600 frames (24 s) because the window's own wander was, after this change, the largest thing left: 7.3 mm at 75 frames against 1.8 mm at 600, for 15 µs of extra median per frame. The cost is that the first seconds of a session run on a coarser size estimate (~9 mm of error after 3 s, ~3 mm after 12 s), which appears as slow drift rather than jitter.

The world landmarks' z is used only for that rigid-size measurement, never for the projected span — it is the weakest thing the network predicts, and a rotation-invariant sum over ten pairs is about the most forgiving use it has.

This is not triangulation. The extra information comes from the network's learned prior on human hand geometry, so depth here is a good inference, not a measurement. Two consequences shape the defaults: it carries a separate, lower gain (--depth-scale, 0.35 against 0.55 for u/v), so a given error moves the arm less than the same error in u or v would; and it is off unless asked for.

Smoothing is a median first, One-Euro second. One-Euro alone is wrong for this signal: its first sample passes through untouched by construction, and that sample arrives exactly when MediaPipe has just acquired the hand and its world landmarks are least reliable. The worst estimate of the session went straight to the output — and again after every dropout, which is the case that matters, because a dropout mid-task kicks an arm that is already engaged. A 7-frame median absorbs those outliers instead of averaging them in, and depth reports nothing at all until the window is full, which downstream holds through rather than treating as a reading.

before after
peak error on acquisition 290 mm 20 mm
worst step on reacquisition 289 mm 0 mm
response to a real 15 cm move 680 ms 440 ms

It is faster despite the extra stage: with outliers handled by the median, the One-Euro floor can sit at the same cutoff as u and v rather than half of it. The cost is ~240 ms of "no depth yet" after the hand is acquired.

More smoothing is not the lever for jitter, which is worth stating because it is the obvious thing to reach for. Measured: halving --depth-cutoff buys about 0.2 mm and costs ~300 ms of lag, and tripling the median window buys as little for as much. What survives the median is not white noise, so low-passing it harder mostly adds delay. The levers that actually work are the two divisions above, and failing that a lower --depth-scale.

Practical notes:

  • --hfov only sets the units of the depth readout. It is one multiplicative constant, and --depth-scale absorbs any error in it, so there is nothing to calibrate — set it if you want the preview overlay to read as real metres.
  • Depth can be unavailable on a frame where the hand tracks fine, and is always unavailable for the first few frames after acquisition. That holds the depth axis rather than failing the sample, so u and v keep flowing.
  • The preview shows the raw per-frame estimate next to the smoothed one, and reports jitter for both. If raw is steady while the reading is not, the smoothing is at fault; if raw itself swings, it is the tracking — more light, or a plainer background.
  • --depth-cutoff smooths depth alone, without touching u and v. See the note above on why it is a weak lever.
  • preview reports depth jitter in millimetres next to the u/v jitter. Hold still and read it: that number decides whether this axis is usable in your lighting, and it degrades faster than u/v do.

Measured on this machine (C930s + RTX 5070 laptop)

camera open → first frame 2.2 s
end-to-end throughput 25.2 Hz (matches the 25 Hz control rate)
MediaPipe inference 20.4 ms mean, 41.6 ms p95 (CPU delegate)
mink IK solve 1.75 ms mean, 4.4 ms p95 (0.63 / 1.5 with --no-avoid-collisions)
hand detection rate 100% of frames, well lit

The IK row is a synthetic sweep at working height, where the table rows are active on every solve; it is the pessimistic case, and both figures come from the same run so the on/off comparison is like for like. Either way MediaPipe is the throughput bottleneck, not the camera or the solver — try --delegate gpu if the loop rate limits you.

Camera notes

teleop/__init__.py sets OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS=0 before cv2 is imported. Without it, opening an external USB camera through MSMF took 175 s to deliver its first frame on this machine; with it, 1.2 s. OpenCV reads the variable at import time, so setting it later has no effect.

DirectShow is not used: it will not negotiate MJPG on the C930s and falls back to YUY2, whose uncompressed 720p30 bitrate exceeds USB 2.0 and throttles the camera to 10 fps. MSMF gets MJPG and runs at 30.

Capture indices shift when USB devices are plugged or unplugged, which is what python -m teleop cameras is for — it also reports whether each device shows a live image or a covered lens.

Tuning

Start with preview, hold your hand still, and read the jitter number. Lower --min-cutoff for a steadier target at the cost of lag; raise --beta if fast motion feels delayed. If the gripper moves the wrong way, press I or O rather than repositioning the camera — the correct sign depends on where the camera sits.

Known limitations

  • Depth is inferred, not measured. --depth leans on MediaPipe's learned prior on hand geometry rather than on triangulation, so it is the first axis to degrade in bad light or heavy occlusion, and its absolute scale varies with how typical your hands are. The clutched relative mapping makes that scale error a gain error rather than an offset, which is why it is tolerable at all. A second camera to the side, with each camera assigned the axes it measures well, is still the honest upgrade — it needs clock-offset estimation plus interpolation onto a common timeline, since two free-running 30 fps streams are up to 16.7 ms apart and a hand at 1 m/s moves 1.7 cm in that window.
  • MediaPipe drops frames. A published AR study measured ~22% no-detection under occlusion against <1% for the transformer-based WiLoR. The clutch is built to tolerate this — a lost hand holds the target rather than releasing — but heavy occlusion will feel bad. WiLoR needs a CUDA GPU and would be the upgrade for the PC-side camera.
  • No demo recording yet. The actions are already in the policy's action space, and build_dataset.py already knows how to write a LeRobotDataset; wiring the teleop loop into it is the remaining step.

Tests

python -m pytest tests -q        # 60 tests, ~5 s, no camera and no GPU needed
  • tests/test_ik_collisions.py — the IK model carries the static world but none of the movable bodies; the collision constraint actually binds (mink states its bound as a velocity while the QP variable is a displacement, which at a 40 ms control period leaves the limit 25× too loose — present, reported, and doing nothing); and it does not bind on anything legal, including the two fingers closing on each other.
  • tests/test_teleop_depth.py — the depth geometry against a synthetic hand at a known distance, and the retargeter's clutch, plane and workspace behaviour.

GPU setup

train_robot_HER_curriculum.py probes the GPU by actually launching a kernel, not just by calling torch.cuda.is_available() — a torch build with no kernels for your GPU's compute capability reports True and then fails at the first real operation. If the probe fails you will see exactly why, and training falls back to CPU:

WARNING: GPU (CUDA) not usable! Training on CPU will be MUCH slower.
Reason: CUDA error: no kernel image is available for execution on the device
  torch 2.5.1+cu121 was built for: sm_50 sm_60 ... sm_90
  this GPU is NVIDIA GeForce RTX 5070 Laptop GPU (sm_120)

The fix is a torch build targeting your GPU's architecture; see pytorch.org/get-started. Newer GPUs (Blackwell / sm_120) need a recent CUDA build. requirements.txt deliberately does not pin torch, because the right build is machine-specific.

Only the main process touches the GPU — SubprocVecEnv workers step MuJoCo on CPU and deliberately skip CUDA initialization, so parallel training does not spawn one CUDA context per worker.

Project structure

panda_env.py                    The environment (mujoco + gymnasium + numpy only)
scripted_expert.py              Phase-based PI/IK expert: eval + demo generation
build_dataset.py                Expert rollouts -> LeRobotDataset
train_act.py                    LeRobot ACT training + Windows checkpoint fix
eval_policy.py                  Held-out evaluation, and --sweep over checkpoints
render_episode.py               Episode -> captioned GIF (expert or policy)
train_robot_HER_curriculum.py   SAC + HER + curriculum training and test rollout
view_panda.py                   Open scene.xml in the MuJoCo viewer
requirements.txt                Sim / RL / teleop deps (torch and lerobot excluded)

tests/
  test_ik_collisions.py         IK collision constraints, kinematics only
  test_teleop_depth.py          Depth geometry + retargeter, no camera needed

teleop/                         Webcam hand teleoperation
  __init__.py                   Sets the MSMF fix before cv2 is imported
  __main__.py                   `python -m teleop {cameras,preview,teleop}`
  camera.py                     Threaded capture, settings measured on this machine
  filters.py                    One-Euro adaptive smoothing filter
  tracker.py                    MediaPipe Hands -> wrist position, pinch, depth
  retarget.py                   Clutched relative map, hand -> TCP target
  ik.py                         mink differential IK on the 'tcp' site,
                                with table and self-collision constraints
  session.py                    The preview and teleop loops
  models/                       hand_landmarker.task            (gitignored)

franka_emika_panda/             Panda MJCF assets
  scene.xml                     <- what everything loads: arm + table + block + goal
  panda.xml, hand.xml           Arm and gripper; 'tcp' site, 'wrist' camera, 'home' key
  assets/                       Meshes

episode.gif                     Rendered sample episode                 (tracked)
panda_her_curriculum_final.zip  SAC policy, predates the env rewrite    (tracked)

data/                           LeRobot datasets                        (gitignored)
outputs/                        ACT runs and checkpoints                (gitignored)
logs/                           Training logs                           (gitignored)
panda_her_tensorboard/          TensorBoard runs, one dir per run       (gitignored)
panda_her_training_state.json   Curriculum stage + step count           (gitignored)
panda_demos*.npz                Flat expert demos                       (gitignored)

Generated files

Everything under data/, outputs/, logs/ and panda_her_tensorboard/ is regenerable and gitignored, along with the curriculum state file, the HER replay buffer, MUJOCO_LOG.TXT, __pycache__/, the MediaPipe weights, and editor and OneDrive noise (desktop.ini).

Two generated artifacts are tracked deliberately: episode.gif, so the README shows something without a clone having to run anything, and panda_her_curriculum_final.zip, so test mode runs on a fresh clone — with the caveat above.

Troubleshooting

ModuleNotFoundError: No module named 'mujoco' / 'lerobot' / 'mediapipe' Wrong environment. lerobot lives only in lerobot_env; mediapipe, mink and stable_baselines3 only in ml_env. See Environments.

OSError: [WinError 1314] A required privilege is not held by the client LeRobot's checkpoint symlink. Launch training through train_act.py, which patches it — do not call LeRobot's train entry point directly on Windows.

An ACT policy scores 0% no matter how long it trains Almost certainly the pre/post-processors. ACTPolicy does not normalise its own inputs or outputs in LeRobot 0.4.4; see Evaluate.

BrokenProcessPool while writing a dataset Video encoding. build_dataset.py already passes parallel_encoding=False and h264; if you changed either, change it back.

RuntimeError: Unable to sample before the end of the first episode learning_starts is below MAX_EPISODE_STEPS * n_envs. safe_learning_starts() handles this automatically; if you bypass it, raise the value.

AssertionError: You must pass an environment when using HerReplayBuffer A HER model cannot be loaded standalone. Pass env= to SAC.load(); the training and test paths already do.

Training is far slower than expected Check the startup banner for the CUDA warning above — the run is probably on CPU.

Renderer / OpenGL errors on Windows Update GPU drivers, or run headless (train mode does not render).

References

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages