Pick-and-place on a Franka Emika Panda in MuJoCo, end to end: a randomised task, a scripted expert that solves it, a LeRobot dataset recorded from that expert, an ACT policy trained on that dataset, and an evaluation that scores the policy against the expert's ceiling on tasks it never saw. Alongside it sit an earlier SAC + HER curriculum track and webcam hand teleoperation that emits the same 8-D action everything else does.
Both build on the same environment, panda_env.py, which depends only on
mujoco, gymnasium and numpy — no learning framework. That separation is
what lets the LeRobot environment, the data-generation workers and the teleop
loop import an env without dragging in stable-baselines3 or firing a CUDA probe
on import.
Imitation learning (the current line). Scripted expert → LeRobot dataset → ACT → evaluation against the expert. Every stage is measured with the same success criterion, so the numbers compare.
SAC + HER + curriculum (the earlier line). train_robot_HER_curriculum.py,
learning from scratch with hindsight relabelling and a three-stage curriculum.
Still runnable; see the note on the checked-in policy.
| File | Role |
|---|---|
panda_env.py |
The environment: task distribution, control law, observations, reward |
scripted_expert.py |
Phase-based PI/IK controller — validates the task, sets the ceiling, records demos |
build_dataset.py |
Rolls the expert out into a LeRobotDataset (video + state + action) |
train_act.py |
Launches LeRobot ACT training, with a Windows checkpoint fix |
eval_policy.py |
Scores a trained policy on held-out seeds; --sweep gives success vs. steps |
render_episode.py |
Renders one episode to a captioned GIF — expert or learned policy |
train_robot_HER_curriculum.py |
SAC + HER + curriculum training and test rollout |
view_panda.py |
Opens the scene in the MuJoCo viewer, no policy |
teleop/ |
Webcam hand teleoperation — see Teleoperation |
tests/ |
pytest suite for the IK collision constraints and the depth geometry |
franka_emika_panda/ |
Panda MJCF; scene.xml is what everything loads |
The pipeline spans two conda environments, because LeRobot and
stable-baselines3 want different torch builds. Both carry mujoco and
gymnasium, so panda_env.py imports in either.
ml_env |
lerobot_env |
|
|---|---|---|
| Interpreter | ...\anaconda3\envs\ml_env\python.exe |
...\anaconda3\envs\lerobot_env\python.exe |
| Python / torch | 3.11.16 / 2.14.0+cu130 | 3.11.16 / 2.10.0+cu128 |
| Also has | stable-baselines3 2.9, mediapipe, mink, Pillow | lerobot 0.4.4 |
| Runs | train_robot_HER_curriculum.py, scripted_expert.py, render_episode.py, python -m teleop, pytest |
build_dataset.py, train_act.py, eval_policy.py |
Do not use the conda
baseenvironment. Its torch 2.5.1+cu121 has no kernels for sm_120:torch.cuda.is_available()returnsTrueand then the first real tensor op fails. See GPU setup.
For a fresh machine:
python -m venv .venv
.venv\Scripts\activate # Windows
# source .venv/bin/activate # macOS/Linux
pip install -r requirements.txt
# then a torch build matching your GPU -- see GPU setup belowrequirements.txt covers the simulation, RL and teleop stack. Two things it
deliberately leaves out: torch, because the right build is machine-specific,
and lerobot, because the imitation-learning stages live in their own
environment (pip install lerobot). render_episode.py also needs Pillow,
which arrives with stable-baselines3[extra].
A red block sits somewhere on the table; a green marker shows where it should end up. Move it there.
Both the block pose and the goal are re-sampled every reset, so a policy faces a distribution of tasks rather than one instance. This was not always true: the env used to restore a single block spawn and never move the goal, which left exactly one goal to learn and nothing for HER's relabelling to do.
| Block spawn | uniform over x ∈ [0.35, 0.65], y ∈ [−0.25, 0.25], resting on the table (z = 0.225) |
| Goal | same xy region; half on the table, half floating 5–20 cm above it |
| Separation | ≥ 10 cm between block and goal, by rejection sampling, so no episode starts solved |
| Success | ‖block − goal‖ < 0.05 m |
Half the goals float on purpose, following the FetchPickAndPlace protocol: a table-only distribution lets a policy collapse onto sliding, an air-only one onto lifting. Keeping both in one distribution forbids either shortcut.
The xy region is smaller than the table (which spans x ∈ [0.25, 0.75], y ∈ [−0.35, 0.35]) so the arm never has to work at the edge of its reach. It was verified reachable: damped-least-squares IK places the TCP within 0.1 mm of every point on a 5×5 grid over it, collision-free.
One env.step holds data.ctrl constant for FRAME_SKIP = 20 physics steps of
2 ms each — a zero-order hold at 25 Hz, which is where manipulation policies
in the literature live. Actions are 7 joint-velocity commands scaled by the
Franka datasheet limits, plus a gripper command:
target_qpos = qpos[:7] + action[:7] * JOINT_VEL_LIMIT * control_dt
Referencing measured qpos rather than the previous target means a blocked
arm cannot wind up an ever-growing position error.
MAX_EPISODE_STEPS = 200 is 8 s of robot time, sized from measurement rather
than taste: the scripted expert needs a median of 81 steps and a p90 of 129 to
finish, so 100 would have truncated most successful trajectories.
Both numbers replaced worse ones. The old env stepped physics once per action at
500 Hz for 1500 steps — 1500 near-identical decisions covering only 3 s of
motion, i.e. less robot time spread over 7.5× more decisions, with a brutal
credit-assignment horizon bought for nothing. It also commanded up to 50 rad/s,
roughly 23× the real joint limit, so the position servo simply saturated against
forcerange and the action had no physically meaningful scale.
Two observation sets, because two kinds of policy consume them.
State (observation, 23-D) — joint positions and velocities, TCP position,
block position, gripper z-axis, plus achieved_goal / desired_goal in
Gymnasium's GoalEnv shape, which is what HER relabels.
Vision — enabled by passing camera_names to the constructor. Adds
pixels/external, pixels/wrist and a 19-D proprio vector: joint positions,
joint velocities, finger apertures, and the TCP position from forward
kinematics. That is only what a real robot reports about itself. The block
position is deliberately absent — it is read straight out of the simulator, no
sensor produces it, and handing it to a vision policy leaks the answer and
produces a number that will not survive contact with a real robot. The policy
has to find the block in the pixels.
Rendering is off by default (~0.6 ms per frame per camera) and the renderer is
built lazily, so constructing an env never opens an OpenGL context it will not
use — SubprocVecEnv workers would otherwise each open one.
The tcp site in panda.xml is the grasp point. Averaging the two finger
body origins, as the env used to, lands ~45 mm behind the fingertip pads, so
the old "gripper position" was never the point that touches anything.
python scripted_expert.py eval --episodes 300 # score it
python scripted_expert.py demos --episodes 500 # -> panda_demos.npz (flat arrays)A six-phase controller — approach, descend, grasp, lift, transport, place — driven by PI resolved-rate IK on the TCP site. It exists for three reasons:
- Validate the environment. Prove the task is solvable within the episode budget before spending GPU-hours on it. A flat RL learning curve on an unsolvable task tells you nothing about your algorithm.
- Provide a ceiling. "SAC reached 60%" means nothing until you know whether the achievable number is 65% or 99%.
- Generate demonstrations. Its (observation, action) pairs are exactly what behaviour cloning, ACT or Diffusion Policy train on.
It scores 99.3% [97.6, 99.8] on 300 held-out tasks (Wilson interval — the normal approximation is wrong this close to 1).
Two implementation details cost the most debugging:
- The outer loop must be PI, not P. The Panda actuators are pure PD position
servos with no gravity compensation, so holding a pose against gravity needs a
standing
ctrl − qposoffset. A proportional controller drives its command to zero as error shrinks, so the arm stalls 1–3 cm short, worst at the far edge of the workspace where it is most extended. The integral term took success from 25% to 88%. - The task is 5-DOF, not 6. Position plus the two rotations that tip the gripper axis downward. Yaw is excluded deliberately: adding it as a zero-target row locks the wrist and wastes a DOF the reach needs. An earlier version pinned yaw to the wrong value and silently commanded a 90° rotation that fought the position servo all episode — 0% success.
Phases 4 and 5 are split for a related reason: driving the TCP straight down onto a table goal at full speed pushes the block into the surface until it pops out of the fingers, which accounted for every remaining failure. Approaching from above and descending at reduced speed took table goals 86.8% → 100%.
# in lerobot_env
python build_dataset.py --episodes 500 --out data/panda_pick_v1Writes a LeRobotDataset directly — no intermediate .npz bridged between two
conda environments.
| Feature | Shape | What it is |
|---|---|---|
observation.images.external |
video, 96×96×3 | third-person RGB, scene context |
observation.images.wrist |
video, 96×96×3 | wrist RGB, what the fingers are doing |
observation.state |
19 | proprioception only (see above) |
observation.environment_state |
3 | goal position |
action |
8 | 7 joint-velocity commands + gripper |
The goal goes in observation.environment_state (LeRobot's FeatureType.ENV,
which ACT consumes via env_state_feature) rather than being concatenated onto
the robot state, because it is a property of the task, not of the robot.
Only successful episodes are recorded — a failed demonstration teaches the wrong
thing. Demo seeds start at 10000, well above the evaluation range, so training
and evaluation tasks can never overlap. The dataset currently in data/ is
490 episodes / 43,794 frames from 500 attempts, at 25 fps.
Two Windows-specific choices are baked in: parallel_encoding=False (LeRobot's
default spawns a ProcessPoolExecutor per episode which, combined with SVT-AV1,
intermittently dies with BrokenProcessPool — a 3-episode smoke test passed, a
500-episode run died on the third) and --vcodec h264 (lighter, and pyav — the
only decoder available here — handles it fastest; LeRobot validates the name
against its own list, so libx264 is rejected).
# in lerobot_env
python train_act.py # defaults below
python train_act.py --steps=40000 --batch_size=16
python train_act.py --policy.chunk_size=100 # override any defaultA thin wrapper around LeRobot's own CLI: everything is passed through, and the defaults are applied only for flags you do not supply.
| Default | Value | Why |
|---|---|---|
--policy.chunk_size / --policy.n_action_steps |
50 | 2 s of committed motion at 25 Hz. LeRobot's default of 100 is 4 s — more than half a median episode, which throws away most of the closed-loop correction chunking exists for |
--save_freq |
2000 | A run killed at step 750 left nothing evaluable with the 5000 default. Checkpoints are ~350 MB; saving 3× more often is cheap insurance |
--num_workers |
12 | Measured: data_s 0.722 → 0.104 → 0.038 at 4/8/12 workers, taking the loop from dataloader-bound to GPU-bound (updt_s 0.143). 1.64 → 2.97 step/s |
--steps |
60000 | |
--batch_size |
16 | |
--output_dir |
outputs/act_v1 |
The Windows fix. LeRobot writes its checkpoints/last pointer with
Path.symlink_to(), which raises OSError: [WinError 1314] A required privilege is not held by the client unless the process is elevated or Developer Mode is
on. It fires inside the checkpoint-saving path, so every save crashes
training: a 300-step probe completed all 300 steps and then died writing the
pointer. train_act.py patches the reference the training script holds and
falls back to a directory junction (mklink /J), which needs no privilege.
Nothing in the installed LeRobot package is modified.
# in lerobot_env
python eval_policy.py outputs/act_v1/checkpoints/last/pretrained_model
python eval_policy.py outputs/act_v1 --sweep --episodes 100Rolls the policy out in the environment the demonstrations came from and reports success with a Wilson interval, printed next to the expert's ceiling.
Two things make that comparison honest, and both are easy to get wrong:
- Identical success criterion. ‖block − goal‖ < 0.05, exactly what the
expert uses. The env also carries
_get_info's curriculum flags, but those belong to the SAC/HER track — stage 1 isgripper_to_block < 0.2. Scoring the policy with one criterion and the expert with another produces two numbers that cannot be compared. - Disjoint seeds. Evaluation uses seeds 0…N−1; demonstrations start at 10000. No evaluated task was ever trained on.
--sweep scores every checkpoint in a run, which is the curve that matters:
loss keeps falling while a policy that never closes the gripper in time stays at
0%.
One inference detail is worth knowing, because getting it wrong scores a hard 0%
no matter how well the network fits. ACTPolicy does not normalise its own
inputs or outputs in LeRobot 0.4.4 — normalize_inputs / unnormalize_outputs
moved out of the model into separate processor pipelines saved beside the
weights. eval_policy.py builds them with make_pre_post_processors (not
PolicyProcessorPipeline.from_pretrained, whose postprocessor operates on a
PolicyAction tensor and raises EnvTransition must be a dictionary. Got Tensor), and runs
obs = preprocessor(obs); a = policy.select_action(obs); a = postprocessor(a).
python render_episode.py --seed 19 # scripted expert
python render_episode.py --seed 19 --policy outputs/act_v1/checkpoints/last/pretrained_model
python render_episode.py --seed 19 --width 640 --height 480 --stride 2Offscreen-renders one frame per control step, captions each with the active phase and the block's lift height, and writes a looping GIF. A success rate tells you a policy fails 30% of the time; a GIF tells you it fails because it closes the gripper 2 cm early. Rendering the failing seeds is usually the more informative choice.
python train_robot_HER_curriculum.py train # parallel envs, headless
python train_robot_HER_curriculum.py train --resume
python train_robot_HER_curriculum.py train-viz # single env, viewer open
python train_robot_HER_curriculum.py test # roll out the saved policy
tensorboard --logdir panda_her_tensorboard| Flag | Meaning |
|---|---|
--timesteps N |
Steps to run this session (default 250000) |
--envs N |
Parallel envs (default: CPU cores − 1, capped at 16) |
--resume |
Continue from panda_her_curriculum_final.zip + saved curriculum stage |
--save-buffer |
Persist the HER replay buffer across runs (correct, but a multi-GB file) |
--episodes N |
Episodes for the test rollout |
--no-test |
Skip the test rollout after training |
Difficulty advances automatically once the success rate over the last 100 episodes exceeds 80%:
| Stage | Goal | Success criterion |
|---|---|---|
| 1 | Move the gripper near the block | gripper within 20 cm of block |
| 2 | Get near the block with the gripper pointed down | within 10 cm and orientation ≥ 0.90 |
| 3 | Move the block to the target | block within 10 cm of goal and orientation ≥ 0.90 |
The reward is a weighted sum of approach, placement and orientation terms,
reweighted per stage in compute_reward.
--resume restores three things that must stay in sync: policy weights from the
.zip, the curriculum stage from panda_her_training_state.json (without
which every resume silently restarts at stage 1), and the step counter, so
TensorBoard curves stay continuous. Training also checkpoints on Ctrl-C, so an
interrupted long run is resumable.
The replay buffer is not in the saved .zip. By default a resumed run
collects RESUME_WARMUP_STEPS fresh transitions before restarting gradient
updates, since otherwise SAC would train against an almost-empty buffer.
--save-buffer carries real experience across runs instead, at the cost of a
large file on disk.
safe_learning_starts() raises learning_starts when needed: HER cannot sample
until at least one episode has finished, and SB3 counts timesteps in units of
n_envs, so the real minimum is MAX_EPISODE_STEPS * n_envs.
panda_her_curriculum_final.zip is tracked so a fresh clone can run test
mode — but it was trained before the environment was rewritten. It comes
from the 1500-step, 500 Hz, qpos + a * dt * 50 control law with a fixed block
spawn and a stationary goal. The network shapes still match, so SAC.load
succeeds and the rollout runs, but the behaviour was learned under different
dynamics on a different task. Treat it as an artifact, not a baseline; retrain
before quoting a number from it.
The stage-1 note also still holds: HER is inert until stage 3. The stage-1
reward uses only the approach term and stage-1 success is
gripper_to_block < 0.2; neither reads desired_goal, so goal relabelling
changes nothing until the placement term is active.
Drive the arm with your hand in front of a webcam — no leader arm, no VR rig.
The teleop/ package tracks one hand, retargets it to a gripper target, solves
IK, and emits the same 8-D action the policies use, so a session here could
become demonstration data without a conversion step.
pip install mediapipe mink
mkdir -p teleop/models
curl -L -o teleop/models/hand_landmarker.task \
https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/latest/hand_landmarker.taskThe 7.8 MB model file is gitignored; HandTracker prints this command if it is
missing.
python -m teleop cameras # which capture index is your webcam today
python -m teleop preview # tracking only, no robot <- start here
python -m teleop teleop # drive the armpreview exists as its own mode deliberately: tracking quality in your
lighting is the assumption everything else rests on, and it costs nothing to
check before a robot is involved.
| Key | Action |
|---|---|
SPACE |
engage / release the clutch |
R |
recenter — re-anchor your hand without moving the gripper |
[ ] |
decrease / increase mapping scale |
D |
toggle the depth axis on / off |
, . |
decrease / increase the depth gain |
I / O / P |
invert the horizontal / vertical / depth axis |
Q or Esc |
quit |
camera ─→ MediaPipe Hands ─→ One-Euro ─→ clutched relative map ─→ mink IK ─→ env.step()
│
└─ image landmarks ÷ world landmarks ─→ depth (optional)
- One model, not two. Hand landmark 0 is the wrist and landmarks 4/8 are
the fingertips, so a single
HandLandmarkeryields both the position signal and the gripper signal. - One-Euro filter, not a plain EMA. It adapts its cutoff to hand speed: heavy smoothing at rest kills the jitter that would make the arm buzz, light smoothing in motion keeps the lag you would otherwise feel while reaching.
- Relative mapping with a clutch. The target is the TCP at engage plus a scaled offset from your hand at engage — so there is no jump on engage, a dropped frame parks the arm instead of throwing it, and re-clutching extends range the way lifting a mouse does.
- mink, not hand-rolled DLS. mink solves a QP per step and enforces joint position and velocity limits inside the solve. Plain damped least squares handles singularities but is blind to joint limits, and a human waving at a camera drives the arm to its limits constantly.
- The table and the arm's own links are constraints, not hopes. The same QP
carries collision rows, so the solver will not command a step that drives the
gripper through the work surface or folds the wrist back into the shoulder.
This is not only about abuse: reaching a perfectly legal target at
(0.5, 0.15, 0.30)transiently dips the fingertips 3.7 mm into the table without it, because differential IK follows the straightest joint-space path and not a straight line in the workspace. Tracking accuracy and settling time are unchanged.--no-avoid-collisionsturns it off. - Plane by default, depth by choice. A single RGB camera measures its two
image axes well and depth badly, so the default drives a plane (
--plane frontalortable) with the third axis pinned and the gripper held pointing at the table.--depthfrees that third axis;Dtoggles it mid-session without moving the gripper. - Depth from the ratio of two landmark sets, not from apparent size. See below — this is the part most likely to be got wrong.
Because ArmIK's velocity limit is exactly the env's JOINT_VEL_LIMIT, the
action derived from the IK solution saturates at exactly 1.0 and is never
silently clipped.
MediaPipe returns two sets of landmarks from one inference: image landmarks in pixels, and world landmarks in metres with the hand's translation removed. The image set is shape × orientation × distance tangled together; the world set is shape × orientation alone. Dividing one by the other cancels shape and orientation and leaves pixels-per-metre, which is a pure function of distance:
distance = focal x hand size x foreshortening ÷ palm span in pixels
where the last two come from the world landmarks and the first from the image landmarks. Foreshortening is what the world landmarks are really for, and it is worth separating from the hand's size — see Two divisions, not one below.
The division is the whole point. Apparent palm width alone is not a depth
measurement — it foreshortens as cos θ when the wrist rolls, and a natural
25° of pronation while reaching forward reads as 10% farther away. Measured
against a synthetic hand at a known 0.60 m (tests/test_teleop_depth.py):
| wrist roll | palm width alone | this method |
|---|---|---|
| 10° | +9 mm | +0.2 mm |
| 25° | +61 mm | +0.6 mm |
| 40° | +179 mm | +1.0 mm |
| 50° | +324 mm | +1.4 mm |
What keeps the residual that small is that every pair among the five palm landmarks contributes, so the near and far halves of a tilted hand average against each other and the weak-perspective error mostly cancels. Pitching the fingers toward the camera is the worse case, peaking near 1.8% (~11 mm) around 40°, because pitch tips out of the image plane the axis the palm is longest along.
Two divisions, not one. The naive version of this divides the image span by the world landmarks' projected span and stops there. That is wrong in a way that only shows up as jitter. The projected span is really two things multiplied — the hand's true size, which is constant, and the foreshortening, which is what we want — and absolute size from a single image is precisely the ambiguous half of the problem, so the network re-guesses it every frame and that guess is noisy. Dividing by it hands the noise straight to the arm.
The same landmarks' full 3D span isolates that guess: pairwise 3D distances are rotation-invariant, so it is size alone. Dividing the projected span by it leaves pure foreshortening and cancels the scale noise, and the hand's size then comes from a long running median — legitimate exactly because a hand's size does not change. Two consequences, both measured:
| world-landmark size noise | one division | two divisions |
|---|---|---|
| 2% | 4.6 mm | 0.9 mm |
| 5% | 10.4 mm | 1.4 mm |
| 10% | 20.3 mm | 2.5 mm |
| 15% | 30.5 mm | 2.9 mm |
At the default gain, 10% size noise is 0.9 mm of gripper jitter rather than 7.1 mm. The size median runs over 600 frames (24 s) because the window's own wander was, after this change, the largest thing left: 7.3 mm at 75 frames against 1.8 mm at 600, for 15 µs of extra median per frame. The cost is that the first seconds of a session run on a coarser size estimate (~9 mm of error after 3 s, ~3 mm after 12 s), which appears as slow drift rather than jitter.
The world landmarks' z is used only for that rigid-size measurement, never
for the projected span — it is the weakest thing the network predicts, and a
rotation-invariant sum over ten pairs is about the most forgiving use it has.
This is not triangulation. The extra information comes from the network's
learned prior on human hand geometry, so depth here is a good inference, not a
measurement. Two consequences shape the defaults: it carries a separate, lower
gain (--depth-scale, 0.35 against 0.55 for u/v), so a given error moves the
arm less than the same error in u or v would; and it is off unless asked for.
Smoothing is a median first, One-Euro second. One-Euro alone is wrong for this signal: its first sample passes through untouched by construction, and that sample arrives exactly when MediaPipe has just acquired the hand and its world landmarks are least reliable. The worst estimate of the session went straight to the output — and again after every dropout, which is the case that matters, because a dropout mid-task kicks an arm that is already engaged. A 7-frame median absorbs those outliers instead of averaging them in, and depth reports nothing at all until the window is full, which downstream holds through rather than treating as a reading.
| before | after | |
|---|---|---|
| peak error on acquisition | 290 mm | 20 mm |
| worst step on reacquisition | 289 mm | 0 mm |
| response to a real 15 cm move | 680 ms | 440 ms |
It is faster despite the extra stage: with outliers handled by the median, the One-Euro floor can sit at the same cutoff as u and v rather than half of it. The cost is ~240 ms of "no depth yet" after the hand is acquired.
More smoothing is not the lever for jitter, which is worth stating because it
is the obvious thing to reach for. Measured: halving --depth-cutoff buys about
0.2 mm and costs ~300 ms of lag, and tripling the median window buys as little
for as much. What survives the median is not white noise, so low-passing it
harder mostly adds delay. The levers that actually work are the two divisions
above, and failing that a lower --depth-scale.
Practical notes:
--hfovonly sets the units of the depth readout. It is one multiplicative constant, and--depth-scaleabsorbs any error in it, so there is nothing to calibrate — set it if you want the preview overlay to read as real metres.- Depth can be unavailable on a frame where the hand tracks fine, and is always
unavailable for the first few frames after acquisition. That holds the depth
axis rather than failing the sample, so
uandvkeep flowing. - The preview shows the raw per-frame estimate next to the smoothed one, and
reports jitter for both. If
rawis steady while the reading is not, the smoothing is at fault; ifrawitself swings, it is the tracking — more light, or a plainer background. --depth-cutoffsmooths depth alone, without touchinguandv. See the note above on why it is a weak lever.previewreports depth jitter in millimetres next to the u/v jitter. Hold still and read it: that number decides whether this axis is usable in your lighting, and it degrades faster than u/v do.
| camera open → first frame | 2.2 s |
| end-to-end throughput | 25.2 Hz (matches the 25 Hz control rate) |
| MediaPipe inference | 20.4 ms mean, 41.6 ms p95 (CPU delegate) |
| mink IK solve | 1.75 ms mean, 4.4 ms p95 (0.63 / 1.5 with --no-avoid-collisions) |
| hand detection rate | 100% of frames, well lit |
The IK row is a synthetic sweep at working height, where the table rows are
active on every solve; it is the pessimistic case, and both figures come from
the same run so the on/off comparison is like for like. Either way MediaPipe is
the throughput bottleneck, not the camera or the solver — try --delegate gpu
if the loop rate limits you.
teleop/__init__.py sets OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS=0 before
cv2 is imported. Without it, opening an external USB camera through MSMF took
175 s to deliver its first frame on this machine; with it, 1.2 s. OpenCV
reads the variable at import time, so setting it later has no effect.
DirectShow is not used: it will not negotiate MJPG on the C930s and falls back to YUY2, whose uncompressed 720p30 bitrate exceeds USB 2.0 and throttles the camera to 10 fps. MSMF gets MJPG and runs at 30.
Capture indices shift when USB devices are plugged or unplugged, which is what
python -m teleop cameras is for — it also reports whether each device shows a
live image or a covered lens.
Start with preview, hold your hand still, and read the jitter number. Lower
--min-cutoff for a steadier target at the cost of lag; raise --beta if fast
motion feels delayed. If the gripper moves the wrong way, press I or O
rather than repositioning the camera — the correct sign depends on where the
camera sits.
- Depth is inferred, not measured.
--depthleans on MediaPipe's learned prior on hand geometry rather than on triangulation, so it is the first axis to degrade in bad light or heavy occlusion, and its absolute scale varies with how typical your hands are. The clutched relative mapping makes that scale error a gain error rather than an offset, which is why it is tolerable at all. A second camera to the side, with each camera assigned the axes it measures well, is still the honest upgrade — it needs clock-offset estimation plus interpolation onto a common timeline, since two free-running 30 fps streams are up to 16.7 ms apart and a hand at 1 m/s moves 1.7 cm in that window. - MediaPipe drops frames. A published AR study measured ~22% no-detection under occlusion against <1% for the transformer-based WiLoR. The clutch is built to tolerate this — a lost hand holds the target rather than releasing — but heavy occlusion will feel bad. WiLoR needs a CUDA GPU and would be the upgrade for the PC-side camera.
- No demo recording yet. The actions are already in the policy's action
space, and
build_dataset.pyalready knows how to write aLeRobotDataset; wiring the teleop loop into it is the remaining step.
python -m pytest tests -q # 60 tests, ~5 s, no camera and no GPU neededtests/test_ik_collisions.py— the IK model carries the static world but none of the movable bodies; the collision constraint actually binds (mink states its bound as a velocity while the QP variable is a displacement, which at a 40 ms control period leaves the limit 25× too loose — present, reported, and doing nothing); and it does not bind on anything legal, including the two fingers closing on each other.tests/test_teleop_depth.py— the depth geometry against a synthetic hand at a known distance, and the retargeter's clutch, plane and workspace behaviour.
train_robot_HER_curriculum.py probes the GPU by actually launching a kernel,
not just by calling torch.cuda.is_available() — a torch build with no kernels
for your GPU's compute capability reports True and then fails at the first
real operation. If the probe fails you will see exactly why, and training falls
back to CPU:
WARNING: GPU (CUDA) not usable! Training on CPU will be MUCH slower.
Reason: CUDA error: no kernel image is available for execution on the device
torch 2.5.1+cu121 was built for: sm_50 sm_60 ... sm_90
this GPU is NVIDIA GeForce RTX 5070 Laptop GPU (sm_120)
The fix is a torch build targeting your GPU's architecture; see
pytorch.org/get-started. Newer GPUs
(Blackwell / sm_120) need a recent CUDA build. requirements.txt deliberately
does not pin torch, because the right build is machine-specific.
Only the main process touches the GPU — SubprocVecEnv workers step MuJoCo on
CPU and deliberately skip CUDA initialization, so parallel training does not
spawn one CUDA context per worker.
panda_env.py The environment (mujoco + gymnasium + numpy only)
scripted_expert.py Phase-based PI/IK expert: eval + demo generation
build_dataset.py Expert rollouts -> LeRobotDataset
train_act.py LeRobot ACT training + Windows checkpoint fix
eval_policy.py Held-out evaluation, and --sweep over checkpoints
render_episode.py Episode -> captioned GIF (expert or policy)
train_robot_HER_curriculum.py SAC + HER + curriculum training and test rollout
view_panda.py Open scene.xml in the MuJoCo viewer
requirements.txt Sim / RL / teleop deps (torch and lerobot excluded)
tests/
test_ik_collisions.py IK collision constraints, kinematics only
test_teleop_depth.py Depth geometry + retargeter, no camera needed
teleop/ Webcam hand teleoperation
__init__.py Sets the MSMF fix before cv2 is imported
__main__.py `python -m teleop {cameras,preview,teleop}`
camera.py Threaded capture, settings measured on this machine
filters.py One-Euro adaptive smoothing filter
tracker.py MediaPipe Hands -> wrist position, pinch, depth
retarget.py Clutched relative map, hand -> TCP target
ik.py mink differential IK on the 'tcp' site,
with table and self-collision constraints
session.py The preview and teleop loops
models/ hand_landmarker.task (gitignored)
franka_emika_panda/ Panda MJCF assets
scene.xml <- what everything loads: arm + table + block + goal
panda.xml, hand.xml Arm and gripper; 'tcp' site, 'wrist' camera, 'home' key
assets/ Meshes
episode.gif Rendered sample episode (tracked)
panda_her_curriculum_final.zip SAC policy, predates the env rewrite (tracked)
data/ LeRobot datasets (gitignored)
outputs/ ACT runs and checkpoints (gitignored)
logs/ Training logs (gitignored)
panda_her_tensorboard/ TensorBoard runs, one dir per run (gitignored)
panda_her_training_state.json Curriculum stage + step count (gitignored)
panda_demos*.npz Flat expert demos (gitignored)
Everything under data/, outputs/, logs/ and panda_her_tensorboard/ is
regenerable and gitignored, along with the curriculum state file, the HER replay
buffer, MUJOCO_LOG.TXT, __pycache__/, the MediaPipe weights, and editor and
OneDrive noise (desktop.ini).
Two generated artifacts are tracked deliberately: episode.gif, so the README
shows something without a clone having to run anything, and
panda_her_curriculum_final.zip, so test mode runs on a fresh clone — with
the caveat above.
ModuleNotFoundError: No module named 'mujoco' / 'lerobot' / 'mediapipe'
Wrong environment. lerobot lives only in lerobot_env; mediapipe, mink
and stable_baselines3 only in ml_env. See Environments.
OSError: [WinError 1314] A required privilege is not held by the client
LeRobot's checkpoint symlink. Launch training through train_act.py, which
patches it — do not call LeRobot's train entry point directly on Windows.
An ACT policy scores 0% no matter how long it trains
Almost certainly the pre/post-processors. ACTPolicy does not normalise its own
inputs or outputs in LeRobot 0.4.4; see Evaluate.
BrokenProcessPool while writing a dataset
Video encoding. build_dataset.py already passes parallel_encoding=False and
h264; if you changed either, change it back.
RuntimeError: Unable to sample before the end of the first episode
learning_starts is below MAX_EPISODE_STEPS * n_envs. safe_learning_starts()
handles this automatically; if you bypass it, raise the value.
AssertionError: You must pass an environment when using HerReplayBuffer
A HER model cannot be loaded standalone. Pass env= to SAC.load(); the
training and test paths already do.
Training is far slower than expected Check the startup banner for the CUDA warning above — the run is probably on CPU.
Renderer / OpenGL errors on Windows
Update GPU drivers, or run headless (train mode does not render).
- MuJoCo Documentation
- LeRobot
- Stable-Baselines3
- mink — differential IK on MuJoCo
- Franka Emika Panda
- Action Chunking with Transformers (ACT): Zhao et al., 2023
- Hindsight Experience Replay: Andrychowicz et al., 2017
- One-Euro filter: Casiez et al., 2012
