Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations
What it is
Pixel-UMI is a robotics research pipeline for learning whole-body mobile manipulation policies from sensorized human demonstrations. It introduces Environment-Aware Retargeting (EAR), which uses reconstructed scene geometry plus demonstrated head and gripper trajectories to produce collision-aware joint and base targets, and Retargeting-Error-Aware Masking (REAM), which uses residual retargeting pose errors during training to down-weight mismatched viewpoints at deployment. On-policy human corrections are then collected via DAgger to capture contact-rich interactions, and the system is evaluated on five tasks with a reported 87.0% average task progress.
The problem it solves
Human demonstration data is scalable but embodiment differences between humans and robots cause retargeting pose errors and viewpoint mismatches, while estimated actions alone fail to capture contact-rich interactions.
What you could use it for
- Training whole-body mobile manipulation policies for robots from human demonstration data
- Retargeting human head/gripper trajectories onto a robot with collision-aware joint and base targets
- Reducing viewpoint mismatch between human demonstrations and robot observations during policy training
- Improving contact-rich bimanual manipulation via on-policy human DAgger corrections
- Benchmarking articulated-object, deformable-object, and constrained-space manipulation tasks
Why it matters
If it works, it would let robot policies be learned from scalable, sensorized human demonstrations despite embodiment differences, lowering the cost of collecting robot-specific whole-body manipulation data and improving performance on contact-rich and collision-constrained tasks.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.