Awesome
bps3D: Batch Rendering for Reinforcement Learning
This repository is the reference implementation of batch rendering, as described in Large Batch Simulation for Deep Reinforcement Learning.
The implementation is relatively scene and task agnostic: both depth sensor and RGB camera outputs are supported for scenes in the GLTF format. Additionally, basic dynamic shading support is available for synthetic scenes.
bps3D can achieve 30,000 to 100,000 FPS on the Gibson and Matterport3D datasets when rendering 64x64 pixel frames.
Dependencies
- CMake
- NVIDIA GPU
- CUDA 10.1 or higher
- NVIDIA driver with Vulkan 1.1 support (440.33 or later confirmed to work)
- Vulkan headers and loader (described below)
The easiest way to obtain the Vulkan dependencies is by installing the latest version of the official Vulkan SDK, available at https://vulkan.lunarg.com/sdk/home. Detailed instructions for installing the SDK are here, specifically the "System Requirements" and "Set up the runtime environment" sections. Be sure that the Vulkan SDK is enabled in your current shell before building.
Building
When using bps3D as the renderer for another CMake-based project, the best integration method is to include this repository as a git submodule, then use CMake's add_subdirectory
on the submodule directory. Targets that depend on the renderer should link against the bps3D
target. An example of this integration is in the bps-nav repository.
A standalone copy of bps3D can be built as follows:
git clone --recursive https://github.com/shacklettbp/bps3D
cd bps3D
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
make
Tools and examples will be built in bps3D/build/bin/
Scene Preprocessing
bps3D requires scenes to be converted into a custom format before rendering. The repository includes a tool for this purpose in bin/preprocess.cpp
that can be used as follows:
./build/bin/preprocess src_file.glb dst_file.bps right up forward texture_dir --texture-dump
This command will convert a source asset, src_file.glb
, into bps3D's custom format: dst_file.bps
. The next three arguments (right
, up
, forward
) describe how the coordinate axes of the source asset are transformed. Finally, texture_dir
specifies that dst_file.bps
will be written to expect textures to be in the texture_dir
directory. The optional --texture-dump
argument tells preprocess
to also extract any textures that are embedded in src_file.glb
to texture_dir
.
The only currently supported input format is GLTF, although asset importing is decoupled from the overall project and can easily be extended. Specific instructions for converting the Gibson and Matterport3D datasets are included in the bps-nav repository.
Texture Compression
bps3D relies on texture compression to reduce memory footprint. All textures must be stored in the KTX2 container, with UASTC compression. Use the toktx
tool from the KTX-Software repository, which has built-in support for common source image formats.
The following is an example command that compresses a JPEG source texture, with rate distortion optimization to reduce on-disk file size:
toktx --uastc --uastc_rdo_q 0.9 --uastc_rdo_d 1024 --zcmp 20 --genmipmap dst.ktx2 src.jpg
Additional Tools
In addition to the preprocess
tool, a handful of other tools are available in the bin/
directory:
fly
: A 3D fly camera for*.bps
files. Depends on OpenGL and GLFW3.singlebench
: Tests renderer performance on a single scene.save_frame
: Test program that renders a batch of RGB and depth outputs for a fixed camera position.
Citation
@article{shacklett21bps,
title = {Large Batch Simulation for Deep Reinforcement Learning},
author = {Brennan Shacklett and Erik Wijmans and Aleksei Petrenko and Manolis Savva and Dhruv Batra and Vladlen Koltun and Kayvon Fatahalian},
journal = {International Conference On Learning Representations (ICLR)},
year = {2021}
}