LeRobot 3D

Easier 3D spatial reasoning for the LeRobot platform.

LeRobot 3D adds 3D grounding to the LeRobot stack. It calibrates multiple depth cameras against the robot, fuses their views into a single point cloud, and tracks the robot’s URDF geometry in the same coordinate frame. The goal is a reusable foundation for 3D-aware robot learning and teleoperation.

Features

• Teleop: drive any number of SO101 leader/follower pairs and Intel RealSense cameras from a single YAML config, no code changes needed.
• Live 3D viewer: a browser UI built on viser shows the fused scene, the robot point cloud and per-link point clouds as they update.
• Calibration: camera extrinsics come from aligning a masked robot point cloud to the URDF mesh, then refining with multi-scale ICP. Intrinsics are saved automatically.
• Dataset recording: episodes are saved as a LeRobotDataset with RGB, lossless depth, intrinsics, extrinsics and per-link robot points for every frame, ready to push to the Hugging Face Hub.
• Extensible API: TeleopSystemConfig, TeleopPointCloudSystem and SystemStateViewer support custom capture scripts.
• Virtual robots: with no hardware attached, it can pose several URDF robots on a grid from action input.

Calibration

💾 View code on GitHub

« back 2 projects