Awesome

 _____ _____ ____  __    _____ _____ 
| __  |   __|    \|  |  |  _  |     |
| __ -|   __|  |  |  |__|     | | | |
|_____|_____|____/|_____|__|__|_|_|_|
 _____  _____  _____  ____   _____  _____ 
| __  ||   __||   | ||    \ |   __|| __  |
|    -||   __|| | | ||  |  ||   __||    -|
|__|__||_____||_|___||____/ |_____||__|__|

BEDLAM Render Tools

This repository contains the render pipeline tools for BEDLAM CVPR2023 paper. It includes automation scripts for SMPL-X data preparation in Blender, data import into Unreal Engine 5 and Unreal rendering.

If you are looking for code to train and evaluate the ML models from the paper then please visit this repository: https://github.com/pixelite1201/BEDLAM

If you are looking for clothing processing code then please visit this repository: https://github.com/PerceivingSystems/bedlam_clothing

Render Pipeline

Data preparation

Data preparation for Unreal (Blender)

• Create animated SMPL-X bodies (v1.1, female/male) from SMPL-X animation data files and export in Alembic ABC format. SMPL-X pose correctives are baked in the Alembic geometry cache and will be used in Unreal without any additional software requirements.
• Details: blender/smplx_anim_to_alembic/

Data import (Unreal)

• Import clothing and SMPL-X Alembic ABC files as GeometryCache
• Import body textures and clothing overlay textures
• Import high-dynamic range panoramic images (HDRIs) for image-based lighting
• Details: unreal/import/

Render sequence generation

BEDLAM Unreal render setup utilizes a data-driven design approach where external data files (be_seq.csv) are used to define the setup of the required Unreal assets for rendering.

• Generate body scene description (be_seq.csv) based on randomization configuration for all the sequences in the desired render job
  • Details: tools/sequence_generation/

Rendering (Unreal)

• Auto-generate Unreal Sequencer LevelSequence assets based on selected body scene description file
• Render generated Sequencer assets with Movie Render Queue using DX12 rasterizer with 7 temporal samples for motion blur
• If depth maps and segmentation masks are desired a second optional render pass will output EXR files (32-bit float, multilayer, cryptomatte) without spatial and temporal samples
• Camera ground truth poses in Unreal coordinates are generated during rendering
• Details: unreal/render/

Post processing

• Generate MP4 movies from image sequences with ffmpeg
• Extract separate depth maps (EXR) and segmentation masks (PNG) if required EXR data is available
• Details: tools/post_render_pipeline/be_post_render_pipeline.sh

Requirements

• Rendering: Unreal Engine 5.0.3 for Windows and good knowledge of how to use it
• Data preparation: Blender (3.2.2 or later)
• Windows (10 or later)
  • Data preparation stage will likely also work under Linux or macOS thanks to Blender but we have not tested this and are not providing support for this option
  • Windows WSL2 subsystem for Linux with Ubuntu 22.04
  • Python for Windows (3.10.2 or later)
• Recommended PC Hardware:
  • CPU: Modern multi-core CPU with high clock speed (Intel i9-12900K)
  • GPU: NVIDIA RTX3090 or higher
  • Memory: 128GB or more
  • Storage: Fast SSD with 8TB of free space

Notes

• GitHub
  • Issues
    • Please check first if your issue was already reported in the issue tracker before opening a new one. Make sure to check both open and also closed issues.
    • Use descriptive name for your issue which clearly states the problem
    • Do not ask several unrelated questions on the same issue
  • Pull requests
    • We are not accepting unrequested pull requests
• Logo: https://github.com/hermanTenuki/ASCII-Generator.site
  • Font: rectangles

Citation

@inproceedings{Black_CVPR_2023,
  title = {{BEDLAM}: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion},
  author = {Black, Michael J. and Patel, Priyanka and Tesch, Joachim and Yang, Jinlong}, 
  booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)},
  pages = {8726-8737},
  month = jun,
  year = {2023},
  month_numeric = {6}
}