Home

Awesome

Omniverse IndoorKit Extension (#ExtendOmniverse 2022 Contest Overall Winner)

teaser

This extension allows to load and record indoor scene tasks for robotics. Introduction video here.

<div> <img src='./img/pickup_AdobeExpress.gif' width='320px'> <img src='./img/drawer_AdobeExpress.gif' width='320px'> <img src='./img/rotate_AdobeExpress.gif' width='320px'> <img src='./img/water_AdobeExpress.gif' width='320px'> </div>

In the field of robotics, it requires a lot of effort to set up even a simple task (e,g. picking up an object) for a robot in a real scene. At present, with the help of Omniverse, not only can we set up tasks for robots in a photo-realistic and physics-reliable manner, but we build this extension to bring high-quality content with a wide range of variability and randomness.

Besides, we design a complete pipeline to load and record the scene, control and replay the robot actions, and render images. We hope this work could encourage academic research in related fields.

<img src="img/ui.png" width="300">

Get started with Omniverse Code/Create [version >= 2022]

Download the release or clone the this repository

Note: The size of the extension including model assets is about 300 MB

git clone https://github.com/yizhouzhao/VRKitchen2.0-IndoorKit

Upzip or locate the root folder as <your-path-to-VRKitchen2.0-IndoorKit>

The file structure of this extension should look like:

<your-path-to-VRKitchen2.0-IndoorKit>
    └───data    [Folder to save the labeling data]
    └───exts    [Omniverse extenstion]
         └───vrkitchen.indoor.kit
                └───asset   [Asset (object, house, e.t.c.) needed for robot tasks]
                └───config  [Extension config]
                └───data    [Extension data]
                └───icons   [Extension icons]
                └───vrkitchen/indoor/kit   [source code]
    └───img   
    └───tool
    │   README.md  
    ......

Add extension to Omniverse

  1. [Open extension manager] After opening Omniverse Code, go to Menu -> Window -> Extension

  2. [Add this extension to Omniverse] Click the <img src="https://github.githubassets.com/images/icons/emoji/unicode/2699.png?v8" width="18"> button, add absolute extension path to Extension Search Paths.

Finally, you can search the vrkitchen.indoor.kit and enable this extension.

Note: The extension path to add is: <your-path-to-VRKitchen2.0-IndoorKit>/exts

add_extension

Play with Indoorkit

The functionality of our Indoorkit has three parts:

1. Task layout

Start with a new stage,

task_layout

The Task layout module allows users to automatically add the task object, robot, and room.

a) Click the task type combo box to select task type from Pick up object, Reorient object, Pour water, and Open drawer.

b) Fill the object id integer field (ranging from 0 to 19), then click the Add object button to automatically add an object and a franka robot to the scene.

Note: Now the robot prim: /World/game/franka is selected, you can change the position and rotation of the robot.

c) Fill the house id integer field (ranging from 0 to 19), then click the Add house button to automatically add a room structure to the task.

Note: Now the robot prim: /World/game/ is automatically selected, you can change the game position and rotation.

d) Click the Record scene button to save the scene information (about task type, object, robot, and room) into a json file.

After recording scene, you can close close the stage without saving.

2. Scene utility

-- Load the recorded scene and change scene features.

task_layout

a) Click New scene to open a new stage (with /World and /World/defaultLight only). This this the same as the command: Menu->File->New...

b) Click Load scene to load the scene from saved information from TASK LAYOUT.

Then you can modify the scene by setting

Note: To load the house material requires users to open the Nucleus server. The materials are from Nucleus.

2. Play

-- play the franka robot.

play_ui

a) Click Record to start playing with the franka robot and recording the robot actions.

To control the robot:

Position the end effector (EE) relative the robot itself, use the Robot control UI or the keyboard:

Rotation the end effector (EE), use the Robot control UI or the keyboard:

To open and close the hand gripper, use the Robot control UI or the keyboard:

b) Click Stop button to stop playing with the franka robot.

c) Click Replay button to replay robot actions.

Note: The Replay and Record are according to the information of task type, object id, and house id.

You may render the desired type of the image at any time playing, replaying or pausing. Click Capture image to get a screenshot.

Finally, you can open the data folders:

path_ui

Cite this work

@article{zhao2022vrkitchen2,
  title={VRKitchen2. 0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse},
  author={Zhao, Yizhou and Gong, Steven and Gao, Xiaofeng and Ai, Wensi and Zhu, Song-Chun},
  journal={arXiv preprint arXiv:2206.11887},
  year={2022}
}

Need more rooms?

Go to this repository: https://github.com/yizhouzhao/VRKitchen2.0-Tutorial

License

Acknowledgement

Thanks to the NVIDIA Academic Hardware Grant Program. Without its general support, this extension could not have possibly been developed so fast and so well.