Home

Awesome

<br> <p align="center"> <h1 align="center"><strong>GRUtopia: Dream General Robots in a City at Scale</strong></h1> <p align="center"> <a href='https://github.com/OpenRobotLab' target='_blank'>OpenRobotLab</a>&emsp; <br> Shanghai AI Laboratory <br> </p> </p> <div id="top" align="center">

arXiv

</div>

🤖 Demo

demo

📋 Contents

🏠 About

<!-- ![Teaser](assets/teaser.jpg) -->

Recent works have been exploring the scaling laws in the field of Embodied AI. Given the prohibitive costs of collecting real-world data, we believe the <b>Simulation-to-Real (Sim2Real) paradigm</b> is a more feasible path for scaling the learning of embodied models.

We introduce project <b>GRUtopia</b> (aka. 桃源 in Chinese), the first simulated interactive 3D society designed for various robots. It features several advancements:

We hope that this work can alleviate the scarcity of high-quality data in this field and provide a more comprehensive assessment of embodied AI research.

🔥 News

📚 Getting Started

Prerequisites

We test our codes under the following environment:

Installation

We provide the installation guide here. You can install locally or use docker and verify the installation easily.

Demo

Following the installation guide, you can verify the installation by running:

python ./GRUtopia/demo/h1_locomotion.py  # start simulation

You can see a humanoid robot (Unitree H1) walking following a pre-defined trajectory in Isaac Sim.

Wander the Demo Scenes

Referring to the guide, you can basically run to wander a demo house:

# python ./GRUtopia/demo/h1_city.py will run a humanoid in the city block
# Its movement is much smaller given the large space of the block.
# Therefore, we recommend try with h1_house.py
python ./GRUtopia/demo/h1_house.py  # start simulation

You can control a humanoid robot to walk around in a demo house and look around from different views by changing the camera view in Isaac Sim (on the top of the UI).

BTW, you can also simply load the demo city USD file into Isaac Sim to freely sightsee the city block with keyboard and mouse operations supported by Omniverse.

WebUI and NPC Demo

Please refer to the guide to try with WebUI and play with NPCs. Note that there are some additional requirements, such as installing with the docker and LLM's API keys.

Documentation & Tutorial

We provide detailed docs and simple tutorials for the basic usage of different modules supported in GRUtopia. Welcome to try and post your suggestions!

📦 Model and Benchmark

Benchmark Overview

<p align="center"> <img src="docs/en/_static/image/benchmark.png" align="center" width="100%"> </p> An embodied agent is expected to actively perceive its environment, engage in dialogue to clarify ambiguous human instructions, and interact with its surroundings to complete tasks. Here, we preliminarily establish three benchmarks for evaluating the capabilities of embodied agents from different aspects: <b>Object Loco-Navigation</b>, <b>Social Loco-Navigation</b>, and <b>Loco-Manipulation</b>. The target object in the instruction are subject to some constraints generated by the world knowledge manager. Navigation paths, dialogues, and actions are depicted in the figure.

For now, please see the paper for more details of our models and benchmarks. We are actively re-organizing the codes and will release them soon. Please stay tuned.

📝 TODO List

🔗 Citation

If you find our work helpful, please cite:

@inproceedings{grutopia,
    title={GRUtopia: Dream General Robots in a City at Scale},
    author={Wang, Hanqing and Chen, Jiahe and Huang, Wensi and Ben, Qingwei and Wang, Tai and Mi, Boyu and Huang, Tao and Zhao, Siheng and Chen, Yilun and Yang, Sizhe and Cao, Peizhou and Yu, Wenye and Ye, Zichao and Li, Jialun and Long, Junfeng and Wang, ZiRui and Wang, Huiling and Zhao, Ying and Tu, Zhongying and Qiao, Yu and Lin, Dahua and Pang Jiangmiao},
    year={2024},
    booktitle={arXiv},
}
</details>

📄 License

GRUtopia's simulation platform is MIT licensed. The open-sourced GRScenes are under the <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License </a><a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/80x15.png" /></a>.

👏 Acknowledgements