Turn multi-view RGB-D video of a human doing a manipulation task into a robot-ready demonstration dataset — no teleoperation rig required.
Human video is cheap and abundant, but naive retargeting from human to robot kinematics produces noisy, jerky trajectories. ViKi captures the demonstration from several calibrated RGB-D cameras, reconstructs the hand in 3D, and solves a whole-trajectory optimisation to land it on a real manipulator.
The Retarget tab replays a recorded episode as a fused point cloud + hand skeleton next to the robot arm converging onto the retargeted trajectory.
ViKi is research code under active development. It is not a finished product, and not everything advertised below is finished.
- The capture → skeleton → retarget → trajectory export path runs end to end.
v0.0.2also adds resolution-independent K4A geometry and a resumable scene-perception branch that produces compact rigid-object models and a separate post-IK object-relative trajectory sidecar. Cube-aware grasp is now the default retarget method for the UR10/Robotiq setup, but requires a tracked cube and has not been validated in simulation. - Accuracy is still being tuned. Triangulation gating, multi-camera sync and
the articulated hand-fit model are the subject of ongoing measurement — see
docs/robust_retarget_experiments.md. There is no ground-truth reference in this setup, so all reported millimetre figures are self-consistency and tracking-error measures, not absolute accuracy. - Replay on hardware is a stub. Nothing exported here has been validated on a physical robot. Do not run a ViKi trajectory on real hardware without your own review and safety checks.
- Exported bundles are unscreened. Because replay is a stub, episodes are exported without a replay verdict. Every manifest says so explicitly.
- Interfaces, defaults and config keys are still moving between commits.
- The repository ships code only. Robot URDFs, model weights, calibration and recordings are not included — see What you must supply yourself.
Multi-view RGB-D capture ← RealSense D435i + Azure Kinect DK (HW-synced)
│
▼
3D hand skeleton ← MediaPipe / RTMPose + multi-view triangulation + depth
│
▼
Smoothing & fusion ← cross-camera fuse, Savitzky–Golay, articulated hand model
│
▼
Robot retargeting ← whole-trajectory IK via PINK / Pinocchio
│
▼
Trajectory bundle ← npz + JSON manifest (works, numpy only)
LeRobot dataset ← ACT / Diffusion Policy (needs optional deps)
Each stage writes a durable artifact into the episode directory, so any stage can be re-run, compared or audited on its own:
raw/ → rec.npz → cln.npz → plan.h5 → (replay.h5) → dataset
└→ SAM masks → semantic cloud → object_models.npz
+ plan.h5 → object_relative.npz → dataset
Everything runs in Docker. The app talks to hardware SDKs (pyrealsense2,
libk4a.so) that live in the image, not on your host.
sudo ./scripts/host_setup.sh # once per host: Docker, udev rules, groups
docker compose up --build # web UI + API on http://localhost:8000Read SETUP_GUIDE.md before plugging in cameras — Azure
Kinect needs a GRUB usbfs bump, xhost +local:, one independent USB host
controller per Kinect, and a sync cable for multi-Kinect capture.
Two Kinects require two independent USB host controllers on this rig. The old single-controller wiring hard-locked the host. The current setup puts one camera on the motherboard Intel xHCI and one on the Renesas PCIe xHCI; dual-camera capture and wired sync now work. Separate sockets or hubs on the same controller are not sufficient. Check the PCI controller topology after rewiring; see SETUP_GUIDE.md. The historical failure is closed in
bugs.mdanddocs/camera_freeze_investigation.md.
| OS | Linux (tested on Ubuntu) |
| Runtime | Docker + Docker Compose |
| GPU | optional for the hand/retarget path; NVIDIA CUDA is required by the scene-perception SAM 2.1 image |
| Cameras | Intel RealSense D435i and/or Azure Kinect DK — one independent USB controller per Kinect |
| Robot | a URDF from robot_descriptions — UR3/UR5/UR10/UR5e/UR10e, iiwa14 (default: ur10) |
You can run the perception stages on pre-recorded episodes with no cameras attached.
This repository contains source code only. data/ and models/ are
gitignored, and nothing below is distributed with ViKi. Most of it is fetched or
produced on first use; some of it you have to go and get.
| What | Where it goes | How you get it |
|---|---|---|
| Robot URDF descriptions | models/robot_descriptions/ |
Downloaded at the first retarget run by the robot_descriptions package. Needs network access from inside the container. |
| Gripper URDFs | models/robot_descriptions/ |
Same mechanism. Only Robotiq 2F-85 is wired up; the other profiles refuse to load and state why (licence/packaging unverified). Supplying your own gripper is on you. |
| Hand-pose model weights | ~/.cache/, models/ |
MediaPipe HandLandmarker and RTMPose ONNX auto-download on first use. The mmpose-heatmap models (HRNetv2, Hourglass-52, SCNet-50, ResNet-50) are not published as ONNX — convert them yourself with mmdeploy or fetch from the OpenMMLab Deploee, then drop the .onnx into models/. |
| SAM 2.1 checkpoint | models/sam2/sam2.1_hiera_small.pt |
Optional, for the scene-perception branch. Download the official Meta checkpoint; ViKi verifies its SHA-256 before inference. SAM 2 code and checkpoints are Apache-2.0 licensed. |
| Camera calibration | data/ |
Produced by you, per rig, with the Calibration tab — intrinsics, extrinsics, world anchor. Calibration is rig-specific and varies between recording sessions; none is shipped. |
| Recordings / episodes | data/episodes/, data/datasets/ |
Yours. Record them with ViKi, or import an existing dataset. |
| Configuration | data/user_configuration.json |
Copied from default_configuration.json on first run, then edited by you (robot, base offset, adapter, profiles). |
| LeRobot export deps | python env | pip install viki[export] — pulls lerobot and torch. The native trajectory bundle needs none of this. |
| Host prerequisites | your machine | GRUB usbcore.usbfs_memory_mb=1000, udev rules, X11 access, USB hubs, sync cable — see SETUP_GUIDE.md. |
Third-party models, URDFs and SDKs carry their own licences, which are not Apache-2.0 and are not granted by this repository. Review them before you redistribute anything you build. Open items are tracked in
docs/licensing_todo.md.
| stage | package | status |
|---|---|---|
| record | viki/cameras |
works |
| calibrate | viki/calibration |
works |
| extract (skeleton) | viki/perception |
works — accuracy tuning ongoing |
| segment (instances + RGB-D lift) | viki/perception |
works as an explicit CUDA stage; missing pixels remain unknown |
| object-model (rigid core/shell) | viki/perception |
required for default cube-aware retarget; built by the explicit CUDA scene stage |
| object-relative (hand/TCP tracks) | viki/object_centric.py |
post-IK diagnostic sidecar; cube-aware IK is a separate geometric pilot |
| prepare (fuse + smooth) | viki/prepare |
works — accuracy tuning ongoing |
| retarget (IK) | viki/retarget |
cube-aware default with readiness checks; geometric pilot only, no simulated grasp verdict |
| replay (hardware validation) | viki/replay |
stub — no hardware validation |
| export → trajectory bundle | viki/export |
works (numpy only) |
| export → LeRobot dataset | viki/export |
partial — needs optional deps; object-relative annotation is a placeholder |
See viki/README.md for the full package map and
cross-cutting rules.
docker compose run --rm cli export data/episodes/<id> --out data/datasets/pickProduces a self-contained bundle — joint trajectory, commanded and achieved tool
pose, gripper command, the human hand pose it came from, and the per-frame
evidence weight — in plain .npz with a JSON manifest that names its units,
frames and provenance. No torch, no parquet, no lerobot.
<out>/dataset.json manifest: schema, units, frames, provenance, index
<out>/episodes/<id>/trajectory.npz the arrays
<out>/episodes/<id>/meta.json that episode's provenance
The manifest's screening field states plainly that nothing was replayed or
vetted. Details in viki/export/README.md.
The web UI (viki/server/static/) is plain HTML/CSS/JS served by FastAPI — no
build step. Tabs: Cameras, Calibration, Record, Extract, Viewer, Retarget,
Export.
The same stages are available headless:
docker compose run --rm cli record ... # capture a synced RGB-D scene
docker compose run --rm cli extract ... # raw/ -> rec.npz
docker compose run --rm sam2 segment ... # optional masks + semantic 3-D cloud
docker compose run --rm sam2 scene <ep> --objects 3 # complete scene branch
docker compose run --rm cli prepare ... # rec.npz -> cln.npz
docker compose run --rm cli retarget ... # cln.npz -> plan.h5
docker compose run --rm cli object-relative <ep> # model + plan -> relative tracks
docker compose run --rm cli export ... # plan.h5 -> dataset
docker compose run --rm cli run ... # extract -> prepare -> retarget -> replay
docker compose run --rm sam2 run <ep> --scene-prompts data/prompts/<scene>.json # labelled scene + coredocker compose run --rm test # full test suite
docker compose run --rm test tests/unit_tests/gripper # a subset (args append to pytest)
docker compose run --rm terminal # bash shell in the containerviki/ is bind-mounted, so code edits apply on container restart — no rebuild
unless pyproject.toml changes.
docs/math.md |
the full mathematical description of the pipeline |
docs/roadmap.md |
current shipped, experimental and remaining work |
docs/releases/v0.0.2.md |
v0.0.2 release scope, measurements and limits |
docs/robust_retarget_experiments.md |
measurement protocols and results (E6–E14) |
docs/object_centric_decision.md |
why production object-driven retarget remains gated |
docs/paper_code_divergences.md |
where the thesis text and the code disagree, and why |
SETUP_GUIDE.md |
hardware bring-up, USB, sync wiring |
ViKi's source code is licensed under Apache-2.0. Third-party dependencies, hardware SDKs, model weights, and robot assets retain their own terms. See the open license audit and remediation checklist before distributing the application or its Docker image.
