Home

Awesome

Leveraging 2D Data to Learn Textured 3D Mesh Generation

This repository contains the original implementation of the above CVPR paper (also available on arXiv). It is a structured VAE that learns a distribution of textured 3D shapes from just 2D images, by learning to explain those images in terms of 3D shapes differentiably rendered over a 2D background. It also includes a mesh parameterisation guaranteed to avoid self-intersections, by having faces push each other out of the way when the shape is deformed.

If this code is useful for your research, please cite us!

@inproceedings{henderson20cvpr,
  title={Leveraging {2D} Data to Learn Textured {3D} Mesh Generation},
  author={Paul Henderson and Vagia Tsiminaki and Christoph Lampert},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020}
}

Prerequisites

git clone --recursive https://github.com/pmh47/textured-mesh-gen

conda install python=3.6 tensorflow-gpu=1.13.1 numpy scipy opencv ffmpeg tqdm
pip install tensorflow-probability==0.6 trimesh meshzoo
pip install --no-deps git+https://github.com/pmh47/dirt

Pushing Mesh Parameterisation

The LP-based mesh-pushing op is independent of the rest of the code, and can be found in src/mesh_intersections. It must be compiled before use.

Compilation

export GUROBI_ROOT=~/packages/gurobi811/linux64

cd src/mesh_intersections
mkdir build && cd build
cmake \
    -DLIBIGL_WITH_CGAL=ON \
    -DLIBIGL_WITH_OPENGL=ON \
    -DLIBIGL_WITH_OPENGL_GLFW=ON \
    -DCMAKE_BUILD_TYPE=Release \
    -DGUROBI_ROOT=$GUROBI_ROOT
    ..
make -j4
cd ..

Troubleshooting

You might need to make some of the following changes, if the build fails or you experience crashes:

Note that it is normal to see occasional warnings about numerical issues at runtime.

Training

We include the full implementation of the structured VAE model described in the paper, as well as code for preprocessing datasets. If you want to use the mesh-pushing parameterisation, follow the steps above first. However, it is quicker and simpler to begin by training with the dense parameterisation.

Datasets

Before training the model, you need to either download one of our preprocessed datasets, or generate this yourself. Preprocessed datasets are available at the following locations:

Each contains a single folder, which should be unzipped into the preprocessed-data folder in this repo.

If you prefer to preprocess the data yourself, we include scripts extract_*_crops.py for doing so. Note that to preprocess the BrnoCompSpeed data, you will need Detectron installed. You should download the original datasets from the following locations:

These should be unzipped into the data folder in this repo before running the relevant preprocessing script.

Training

The following commands may be run from the src folder to reproduce (up to stochastic variability and small bug-fixes) the models used in the paper, in setting mask with parameterisation dense. Images will be written to output/images at regular intervals.

To use setting no-mask instead, add with-gt-masks=0 to any of the above. To use parameterisation pushing instead, add shape-model=VAE-seq-att-pushing.