Home

Awesome

Sparse-to-Dense

This repo implements the training and testing of deep regression neural networks for "Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image" by Fangchang Ma and Sertac Karaman at MIT. A video demonstration is available on YouTube. This repo offers the original implementation of the paper in Torch. The PyTorch version can be found here.

<p align="center"> <img src="http://www.mit.edu/~fcma/images/ICRA2018.png" alt="photo not available" width="50%" height="50%"> <img src="https://j.gifs.com/Z4qDow.gif" alt="photo not available" height="50%"> </p>

This repo can be used for training and testing of

Contents

  1. Requirements
  2. Training
  3. Testing
  4. Trained Models
  5. Benchmark
  6. Citation

Requirements

See the installation instructions for a step-by-step guide.

Training

The training scripts come with several options, which can be listed with the --help flag.

th main.lua --help

To run the training, simply run main.lua. By default, the script runs the RGB-based prediction network on NYU-Depth-V2 with 1 GPU and 2 data-loader threads without using pretrained weights.

th main.lua 

To train networks with different datasets, input modalities, loss functions, and components, see the example below:

th main.lua -dataset kitti -inputType rgbd -nSample 100 -criterion l1 -encoderType conv -decoderType upproj -pretrain true

Training results will be saved under the results folder.

Model Options

ParameterOptionsRemarks
datasetsnyudepthv2, kitti
inputTypergb, rgbd, d, g, gdd:sparse depth only; g: grayscale
nSamplenon-negative integer (0 for rgb and g)
criterionl1, l2, berhu
pretrainfalse, true
replinear, log, inverserepresentation of input depth
encoderTypeconv, depthsep, channeldropdepthsep: depthwise separable convolution
decoderTypeupproj, upconv, deconv2, deconv3deconv_n: transposed convolution with kernel size n-by-n

Testing

To test the performance of a trained model, simply run main.lua with the -testOnly true option, along with other model options. For instance,

th main.lua -testOnly true -dataset kitti -inputType rgbd -nSample 100 -criterion l1 -encoderType conv -decoderType upproj -pretrain true

Trained Models

Download our trained models at http://datasets.lids.mit.edu/sparse-to-dense/results/ to the results folder. For instance,

cd results
wget -r -np -nH --cut-dirs=2 --reject "index.html*" http://datasets.lids.mit.edu/sparse-to-dense/results/nyudepthv2.input=rgbd.nsample=200.rep=linear.encoder=conv.decoder=upproj.criterion=l1.lr=0.01.bs=16.pretrained=true/
cd ..

More trained models will be released.

Benchmark

Citation

If you use our code or method in your work, please consider citing the following:

@article{Ma2017SparseToDense,
	title={Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image},
	author={Ma, Fangchang and Karaman, Sertac},
	booktitle={ICRA},
	year={2018}
}
@article{ma2018self,
	title={Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera},
	author={Ma, Fangchang and Cavalheiro, Guilherme Venturelli and Karaman, Sertac},
	journal={arXiv preprint arXiv:1807.00275},
	year={2018}
}

Please direct any questions to Fangchang Ma at fcma@mit.edu.