Awesome

The RobotriX

Enter the RobotriX, an extremely photorealistic indoor dataset designed to enable the application of deep learning techniques to a wide variety of robotic vision problems. The RobotriX consists of hyperrealistic indoor scenes which are explored by robot agents which also interact with objects in a visually realistic manner in that simulated world. Photorealistic scenes and robots are rendered by Unreal Engine into a virtual reality headset which captures gaze so that a human operator can move the robot and use controllers for the robotic hands; scene information is dumped on a per-frame basis so that it can be reproduced offline using UnrealCV to generate raw data and ground truth labels. By taking this approach we were able to generate a dataset of 38 semantic classes across 512 sequences totaling 8M stills recorded at +60 frames per second with full HD resolution. For each frame, RGB-D and 3D information is provided with full annotations in both spaces. Thanks to the high quality and quantity of both raw information and annotations, the RobotriX will serve as a new milestone for investigating 2D and 3D robotic vision tasks with large-scale data-driven techniques.

Contents

1. Data
2. UnrealROX
3. Assets
4. Troubleshooting
5. License
6. Contact

Data

We generated a dataset of 512 sequences recorded on 16 different indoor layouts at +60 FPS with a duration that spans between one and five minutes each. That adds up to a total of approximately eight million individual frames. This initial release of the dataset contains 32 detection and 39 semantic classes. The categories were selected from the most common and useful household goods in indoor environments for social robots.

Type	00 <br>	01 <br>	02 <br>	03 <br>	04 <br>	05 <br>	06 <br>	07 <br>	08 <br>	09 <br>	10 <br>	11 <br>	12 <br>
Semantic	void	wall	floor	ceiling	window	door	table	chair	lamp	sofa	cupboard	screen	hand
Detection	-	-	-	-	-	-	table	chair	lamp	sofa	cupboard	screen	hand

Type	13 <br>	14 <br>	15 <br>	16 <br>	17 <br>	18 <br>	19 <br>	20 <br>	21 <br>	22 <br>	23 <br>	24 <br>	25 <br>
Semantic	frame	bed	fridge	whiteboard	book	bottle	plant	furniture	toilet	phone	bathtub	cup	mat
Detection	frame	bed	fridge	whiteboard	book	bottle	plant	-	lamp	toilet	phone	bathtub	cup

Type	26 <br>	27 <br>	28 <br>	29 <br>	30 <br>	31 <br>	32 <br>	33 <br>	34 <br>	35 <br>	36 <br>	37 <br>	38 <br>
Semantic	mirror	sink	box	mouse	keyboard	bin	cushion	shelf	bag	curtain	kitchen_stuff	bath_stuff	prop
Detection	mirror	sink	box	mouse	keyboard	bin	cushion	shelf	bag	-	kitchen_stuff	bath_stuff	prop

Due to the huge size of the data (~7 TiB), we are currently distributing part of it via private links to our FTP server to avoid excessive traffic (drop a mail to agarcia@dtic.ua.es for them). However, half of the dataset is already available (and increasing every day) through OSF at https://osf.io/b3g2y/ (OSF is a free, open source web application that connects and supports the research workflow, enabling scientists to increase the efficiency and effectiveness of their research. Researchers use OSF to collaborate, document, archive, share, and register research projects, materials, and data. OSF is the flagship product of the non-profit Center for Open Science).

IMPORTANT: Instructions to unzip data from OSF

The data uploaded to OSF was zipped in chunks. For example, suppose we are trying to unzip rgb files of a scene, so you have rgb.z01, rgb.z02, rgb.z03... and rgb.zip main file. In order to unzip the data you can do the following:

1. Use 7-zip software which should be working and unzip rgb.zip.
2. Use Linux unzip, but firstly you should:

• Put all the parts together doing the following: zip -F rgb.zip --out rgb_full.zip
• Then you can proceed and unzip the big file rgb_full.zip doing: unzip rgb_full.zip

If you don't follow the instructions above you probably get bad zipfile offset errors.

The following data is available at the OSF project page:

Sequences

Each sequence is recorded as a TXT file that describes it and allows its offline playback to generate the aforementioned data. As a matter of fact, those TXT files are processed and converted into JSON files for improved readability and to make them easier to parse. Those sequence descriptor files contain all the information needed to generate the images and extract ground truth.

Raw Data

For each frame, we provide the following data:

• 3D poses for the cameras, objects, and robot joints.
• RGB image at 1920x1080 resolution in 24-bit JPG(95%) format (instead of PNG for reduced size).
• Depth map at 1920x1080 resolution in 16-bit grayscale PNG format.
• 2D instance mask at 1920x1080 resolution in RGB 24-bit PNG format.

Ground Truth

For each frame, we provide the tools to generate the following annotations:

• 2D class mask at 1920x1080 resolution in RGB 24-bit PNG format.
• 2D/3D object instance bounding boxes in XML format.
• 3D point cloud in PLY format with RGB color.
• 3D instance/class mask in PLY format with RGB color.

Due to the excessive size of that large-scale ground truth, we provide the needed tools and instructions so that anyone can generate the annotations they need locally using the aforementioned raw data.

UnrealROX

IMPORTANT

The data for this paper was generated using a deprecated tool which extended UnrealCV to generate all the data we needed. This tool has been superseded by UnrealROX, a compatible and home-brewed C++ solution for UnrealEngine that allows efficient and flexible data recording in virtual reality and offline generation with annotations. Since this tool is better suited for our purposes (more efficient, flexible, and complete), we have removed all previous references in this repository to old RobotriX tools that we mention in the paper (although they can be obtained through commit history).

UnrealROX is described in detail at the following arXiv paper and it is released at the following GitHub/3dperceptionlab/unrealrox repository.

IMPORTANT

Assets

Assets for this project are originated from two sources: UE4Arch and Unreal Engine Marketplace so they can be acquired from there. At this moment, we are still in conversations with both parties to release the modified assets as we used them in our scenes.

Troubleshooting

For any kind of problems, configurations, or instructions, please read carefully UnrealROX's documentation.

We encourage any user to submit any issue related to the data itself using GitHub's built-in issue system within this repository. For any other issue related to the data generation process or tool, please submit the adequate issue to UnrealROX's repository. Improvements and critics are welcome at all fronts!

License

Both the data for The RobotriX and the code for UnrealROX are released under the MIT license.

Citation

If you use this dataset or the UnrealROX tool, please cite:

• Garcia-Garcia, A., Martinez-Gonzalez, P., Oprea, S., Castro-Vargas, J. A., Orts-Escolano, S., Garcia-Rodriguez, J., & Jover-Alvarez, A. (2018, October). The RobotriX: An Extremely Photorealistic and Very-Large-Scale Indoor Dataset of Sequences with Robot Trajectories and Interactions. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 6790-6797). IEEE.

@inproceedings{garcia2018robotrix,
  title={The RobotriX: An Extremely Photorealistic and Very-Large-Scale Indoor Dataset of Sequences with Robot Trajectories and Interactions},
  author={Garcia-Garcia, Alberto and Martinez-Gonzalez, Pablo and Oprea, Sergiu and Castro-Vargas, John Alejandro and Orts-Escolano, Sergio and Garcia-Rodriguez, Jose and Jover-Alvarez, Alvaro},
  booktitle={2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={6790--6797},
  year={2018},
  organization={IEEE}
}

• Martinez-Gonzalez, P., Oprea, S., Garcia-Garcia, A., Jover-Alvarez, A., Orts-Escolano, S., & Garcia-Rodriguez, J. (2018). UnrealROX: An eXtremely Photorealistic Virtual Reality Environment for Robotics Simulations and Synthetic Data Generation. arXiv preprint arXiv:1810.06936.

@article{martinez2018unrealrox,
  title={UnrealROX: An eXtremely Photorealistic Virtual Reality Environment for Robotics Simulations and Synthetic Data Generation},
  author={Martinez-Gonzalez, Pablo and Oprea, Sergiu and Garcia-Garcia, Alberto and Jover-Alvarez, Alvaro and Orts-Escolano, Sergio and Garcia-Rodriguez, Jose},
  journal={arXiv preprint arXiv:1810.06936},
  year={2018}
}

Contact / Authors

Please contact the authors if you have any questions or requests.

• Alberto Garcia-Garcia [Design, Prototyping, Data Generation, Project Lead] (agarcia@dtic.ua.es)
• Pablo Martinez-Gonzalez [Design, UE4 Backend, Lead Programmer] (pmartinez@dtic.ua.es)
• Sergiu Oprea [Grasping, Data Generation] (soprea@dtic.ua.es)
• John A. Castro-Vargas [Support Programmer, Data Generation] (jacastro@dtic.ua.es)
• Alvaro Jover-Alvarez [UE4 Expert, Support Programmer] (ajover@dtic.ua.es)
• Sergio Orts-Escolano [Design, Technical Advice] (sorts@ua.es)
• Jose Garcia-Rodriguez [Technical Advice] (jgarcia@dtic.ua.es)