Awesome

Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation

Mingyo Seo, Steve Han, Kyutae Sim, Seung Hyeon Bang, Carlos Gonzalez, Luis Sentis, Yuke Zhu

Project | arXiv

Abstract

We tackle the problem of developing humanoid loco-manipulation skills with deep imitation learning. The challenge of collecting human demonstrations for humanoids, in conjunction with the difficulty of policy training under a high degree of freedom, presents substantial challenges. We introduce TRILL, a data-efficient framework for learning humanoid loco-manipulation policies from human demonstrations. In this framework, we collect human demonstration data through an intuitive Virtual Reality (VR) interface. We employ the whole-body control formulation to transform task-space commands from human operators into the robot's joint-torque actuation while stabilizing its dynamics. By employing high-level action abstractions tailored for humanoid robots, our method can efficiently learn complex loco-manipulation skills. We demonstrate the effectiveness of TRILL in simulation and on a real-world robot for performing various types of tasks.

If you find our work useful in your research, please consider citing.

Dependencies

• Python 3.8.5 (recommended)
• Robosuite 1.4.0
• Robomimic 0.2.0
• PyTorch

Usage

Please see Getting Started.

Dataset and pre-trained models

We provide our demonstration dataset in the door simulation environment (link) and trained models of the Visuomotor Policies (link). We also plan to open our demonstration dataset and trained models in the workbench simulation environment in the near future.

Implementation Details

Please see this page for detailed information on our implementation, including the whole-body controller, model architecture, and teleoperation system.

Related repository

The implementation of the whole-body control is based on PyPnC.

Citing

@inproceedings{seo2023trill,
   title={Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation},
   author={Seo, Mingyo and Han, Steve and Sim, Kyutae and 
           Bang, Seung Hyeon and Gonzalez, Carlos and 
           Sentis, Luis and Zhu, Yuke},
   booktitle={IEEE-RAS International Conference on Humanoid Robots (Humanoids)},
   year={2023}
}