Awesome
MeshSDF
This is a PyTorch implementation of the NeurIPS'20 SpotLight paper "MeshSDF: Differentiable Iso-Surface Extraction". You can find the paper here.
Installation
To get started, simply clone the repo and run the setup bash script, which will take care of installing all packages and dependencies.
git clone https://github.com/cvlab-epfl/MeshSDF.git
cd MeshSDF
./setup.sh
Data
In our project, we store data according to the following structure:
data/
<dataset_name>/
samples/
<instance_name>.npz
meshes/
<instance_name>.obj
renders/
<instance_name>/
00.png
...
rendering_metadata.txt
We provide pre-processed and subsampled ShapeNet data for cars and chairs to get you started.
Simply download it and unzip it in the data/
folder to get going.
In case you are interested in using different data, below we detail how to obtain and pre-process the ShapeNet data we used in our project. Do not forget to cite the authors of the respective papers if you find this data useful for your research.
Mesh data
We use data from ShapeNetCore.v1. Specifically, we use cleaned surface meshes from here. Please refer to their repo to download ShapeNetCore.v1 clean models.
SDF data
Once you have downloaded cleaned ShapeNetCore.v1 models, we generate SDF training data using the pre-processing script from DeepSDF. Please refer to their repo for more info on how to generate SDF samples.
Image data
We use rendered ShapeNetCore.v1 models from DISN. Please refer to their repo for more info on how to generate synthetic renders.
Differentiable iso-surface extraction
We show how to backpropagate gradient information from mesh vertices to latent vector while modifying surface mesh topology through a simple toy-experiment.
As an example of a loss function defined on an explicit surface mesh, we consider silhouette distance, obtained through differentiable rasterization. For this experiment, we consider a dataset consisting of two shapes, bob, a genus-1 rubber duck, and spot, a genus-0 cow.
To get started, first learn a deep signed distance function representation for this dataset by running
python train_deep_sdf.py -e experiments/bob_and_spot
You can then exploit our differentiability result to minimize silhouette distance between spot and bob
python demo_optimizer.py -e experiments/bob_and_spot/
In our paper, we also propose an heuristic to accelerate iso-surface extraction when performing optimization in an iterative fashion, which results in a noticeable speed-up. You can use our heuristic by running
python demo_optimizer.py -e experiments/bob_and_spot/ --fast
Single-view reconstruction
You can train a SVR model for cars by running
python train_svr.py -e experiments/cars_svr/
Once the model is trained, you can run silhouette based refinement, as presented in our work, by running
python reconstruct_svr.py -e experiments/cars_svr/
Finally, to gather results and compute metrics, run
python metrics_svr.py -e experiments/cars_svr/
Cite us
If you find MeshSDF useful for your research, please cite our paper:
@inproceedings{remelli2020meshsdf,
author = {Remelli, Edoardo and Lukoianov, Artem and Richter, Stephan and Guillard, Benoit and Bagautdinov, Timur and Baque, Pierre and Fua, Pascal},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
pages = {22468--22478},
publisher = {Curran Associates, Inc.},
title = {MeshSDF: Differentiable Iso-Surface Extraction},
url = {https://proceedings.neurips.cc/paper/2020/file/fe40fb944ee700392ed51bfe84dd4e3d-Paper.pdf},
volume = {33},
year = {2020}
}