Home

Awesome

MarrNet: 3D Shape Reconstruction via 2.5D Sketches

This repository contains pre-trained models and testing code for MarrNet presented at NIPS 2017.

http://marrnet.csail.mit.edu

<table> <tr> <td><img src="http://marrnet.csail.mit.edu/repo/input/chair_1.png" width="210"></td> <td><img src="http://marrnet.csail.mit.edu/repo/output/chair_1.png" width="210"></td> <td><img src="http://marrnet.csail.mit.edu/repo/input/chair_2.png" width="210"></td> <td><img src="http://marrnet.csail.mit.edu/repo/output/chair_2.png" width="210"></td> <td><img src="http://marrnet.csail.mit.edu/repo/input/chair_3.png" width="210"></td> <td><img src="http://marrnet.csail.mit.edu/repo/output/chair_3.png" width="210"></td> </tr> </table>

Prerequisites

Torch

We use Torch 7 (http://torch.ch) for our implementation with these additional packages:

Visualization

Installation

Our current release has been tested on Ubuntu 14.04.

Cloning the repository

git clone git@github.com:jiajunwu/marrnet.git

Downloading pretrained models (920M)

cd marrnet
./download_models.sh

Guide

3D shape reconstruction (main.lua)

We show how to reconstruct shapes from 2D images by using our pre-trained models. The file (main.lua) has the following options.

Usage

th main.lua -imgname chair_1.png

The output is saved under folder ./output, named %imgname.mat, where %imgname is the name of the image without the filename extension.

Note that the input image will be automatically resized to 256 x 256. The model works the best if the chair is centered and fills up about half of the whole image (like image/chair_1.png).

Visualization

We offer two ways of visualizing results, one in MATLAB and the other in Blender. We used the Blender visualization in our paper. The MATLAB visualization is easier to install and run, but it is also slower, and its output has a lower quality compared with Blender.

MATLAB: Please use the function visualization/matlab/visualize.m for visualization. The MATLAB code allows users to either display rendered objects or save them as images. The script also supports downsampling and thresholding for faster rendering. The color of voxels represents the confidence value.

Options include

Usage (after running th main.lua -imgname chair_1.png, in MATLAB, in folder visualization/matlab):

visualize('../../output/chair_1.mat', 0, 2, 0.03, 'chair_1')

The code will visualize the shape in 2 different views. The visualization might take a while. Please change the step parameter if it is too slow.

Blender: Options for the Blender visualization include

Usage (after running th main.lua -imgname chair_1.png):

cd visualization/blender
blender --background --python render.py -- ../../output/chair_1.mat ../result/ chair_1

Blender will render 3 different views, where the first view should be the same as the input image.

Reference

@inproceedings{marrnet,
  title={{MarrNet: 3D Shape Reconstruction via 2.5D Sketches}},
  author={Wu, Jiajun and Wang, Yifan and Xue, Tianfan and Sun, Xingyuan and Freeman, William T and Tenenbaum, Joshua B},
  booktitle={Advances In Neural Information Processing Systems},
  year={2017}
}

For any questions, please contact Jiajun Wu (jiajunwu@mit.edu) and Yifan Wang (wangyifan1995@gmail.com).