Awesome
Omniverse IndoorKit Extension (#ExtendOmniverse 2022 Contest Overall Winner)
This extension allows to load and record indoor scene tasks for robotics. Introduction video here.
<div> <img src='./img/pickup_AdobeExpress.gif' width='320px'> <img src='./img/drawer_AdobeExpress.gif' width='320px'> <img src='./img/rotate_AdobeExpress.gif' width='320px'> <img src='./img/water_AdobeExpress.gif' width='320px'> </div>In the field of robotics, it requires a lot of effort to set up even a simple task (e,g. picking up an object) for a robot in a real scene. At present, with the help of Omniverse, not only can we set up tasks for robots in a photo-realistic and physics-reliable manner, but we build this extension to bring high-quality content with a wide range of variability and randomness.
Besides, we design a complete pipeline to load and record the scene, control and replay the robot actions, and render images. We hope this work could encourage academic research in related fields.
<img src="img/ui.png" width="300">Get started with Omniverse Code/Create [version >= 2022]
Download the release or clone the this repository
Note: The size of the extension including model assets is about 300 MB
git clone https://github.com/yizhouzhao/VRKitchen2.0-IndoorKit
Upzip or locate the root folder as <your-path-to-VRKitchen2.0-IndoorKit>
The file structure of this extension should look like:
<your-path-to-VRKitchen2.0-IndoorKit>
└───data [Folder to save the labeling data]
└───exts [Omniverse extenstion]
└───vrkitchen.indoor.kit
└───asset [Asset (object, house, e.t.c.) needed for robot tasks]
└───config [Extension config]
└───data [Extension data]
└───icons [Extension icons]
└───vrkitchen/indoor/kit [source code]
└───img
└───tool
│ README.md
......
Add extension to Omniverse
-
[Open extension manager] After opening Omniverse Code, go to
Menu
->Window
->Extension
-
[Add this extension to Omniverse] Click the <img src="https://github.githubassets.com/images/icons/emoji/unicode/2699.png?v8" width="18"> button, add absolute extension path to
Extension Search Paths
.
Finally, you can search the vrkitchen.indoor.kit
and enable this extension.
Note: The extension path to add is:
<your-path-to-VRKitchen2.0-IndoorKit>/exts
Play with Indoorkit
The functionality of our Indoorkit has three parts:
- TASK LAYOUT: to set up object, robot, and room for one task.
- SCENE UTILITY: to load scene and set up scene features including light, sky, matrial, and e.t.c.
- PLAY: to control the robot to perform and the task.
1. Task layout
Start with a new stage,
The Task layout
module allows users to automatically add the task object, robot, and room.
a) Click the task type combo box
to select task type from Pick up object, Reorient object, Pour water, and Open drawer.
b) Fill the object id integer field
(ranging from 0 to 19), then click the Add object
button to automatically add an object and a franka robot to the scene.
Note: Now the robot prim:
/World/game/franka
is selected, you can change the position and rotation of the robot.
c) Fill the house id integer field
(ranging from 0 to 19), then click the Add house
button to automatically add a room structure to the task.
Note: Now the robot prim:
/World/game/
is automatically selected, you can change the game position and rotation.
d) Click the Record scene
button to save the scene information (about task type, object, robot, and room) into a json file.
After recording scene, you can close close the stage without saving.
2. Scene utility
-- Load the recorded scene and change scene features.
a) Click New scene
to open a new stage (with /World and /World/defaultLight only). This this the same as the command: Menu
->File
->New...
b) Click Load scene
to load the scene from saved information from TASK LAYOUT
.
Then you can modify the scene by setting
Visible ground
: show ground planeLight intensity
: change the defaultLight intensitySky type
: change the sky backgroundRandom house material
: change floor and wall material
Note: To load the house material requires users to open the
Nucleus
server. The materials are fromNucleus
.
Enable isosurface
: enable isosurface option for water tasks.
2. Play
-- play the franka robot.
a) Click Record
to start playing with the franka robot and recording the robot actions.
To control the robot:
Position the end effector (EE) relative the robot itself, use the Robot control
UI or the keyboard:
- [W] Move EE forward;
- [S] Move EE backward;
- [A] Move EE to the left;
- [D] Move EE to the right
- [E] Move EE upward;
- [D] Move EE downward.
Rotation the end effector (EE), use the Robot control
UI or the keyboard:
- [ARROW UP] Rotate EE upward;
- [ARROW DOWN] Rotate EE downward;
- [ARROW LEFT] Rotate EE to the left;
- [ARROW RIGHT] Rotate EE to the right.
To open and close the hand gripper, use the Robot control
UI or the keyboard:
- [LEFT CONTROL] Open/Close the gripper.
b) Click Stop
button to stop playing with the franka robot.
c) Click Replay
button to replay robot actions.
Note: The
Replay
andRecord
are according to the information oftask type
,object id
, andhouse id
.
You may render the desired type of the image at any time playing, replaying or pausing. Click Capture image
to get a screenshot.
Finally, you can open the data folders:
Cite this work
@article{zhao2022vrkitchen2,
title={VRKitchen2. 0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse},
author={Zhao, Yizhou and Gong, Steven and Gao, Xiaofeng and Ai, Wensi and Zhu, Song-Chun},
journal={arXiv preprint arXiv:2206.11887},
year={2022}
}
Need more rooms?
Go to this repository: https://github.com/yizhouzhao/VRKitchen2.0-Tutorial
License
-
The rooms in this repository are from Trescope, under the MIT License
-
The drawers and bottles in this repository are from SAPIEN, under this Term of Use
-
The cups in this repository are from AI2THOR, under the Apache License.
-
This repository is for OMNIVERSE CODE CONTEST, under the OMNIVERSE PUBLISHING AGREEMENT .
Acknowledgement
Thanks to the NVIDIA Academic Hardware Grant Program. Without its general support, this extension could not have possibly been developed so fast and so well.