Home

Awesome

sparse-to-dense.pytorch

This repo implements the training and testing of deep regression neural networks for "Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image" by Fangchang Ma and Sertac Karaman at MIT. A video demonstration is available on YouTube.

<p align="center"> <img src="http://www.mit.edu/~fcma/images/ICRA2018.png" alt="photo not available" width="50%" height="50%"> <img src="https://j.gifs.com/Z4qDow.gif" alt="photo not available" height="50%"> </p>

This repo can be used for training and testing of

The original Torch implementation of the paper can be found here.

Contents

  1. Requirements
  2. Training
  3. Testing
  4. Trained Models
  5. Benchmark
  6. Citation

Requirements

This code was tested with Python 3 and PyTorch 0.4.0.

Training

The training scripts come with several options, which can be listed with the --help flag.

python3 main.py --help

For instance, run the following command to train a network with ResNet50 as the encoder, deconvolutions of kernel size 3 as the decoder, and both RGB and 100 random sparse depth samples as the input to the network.

python3 main.py -a resnet50 -d deconv3 -m rgbd -s 100 --data nyudepthv2

Training results will be saved under the results folder. To resume a previous training, run

python3 main.py --resume [path_to_previous_model]

Testing

To test the performance of a trained model without training, simply run main.py with the -e option. For instance,

python3 main.py --evaluate [path_to_trained_model]

Trained Models

A number of trained models is available here.

Benchmark

The following numbers are from the original Torch repo.

Citation

If you use our code or method in your work, please consider citing the following:

@article{Ma2017SparseToDense,
	title={Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image},
	author={Ma, Fangchang and Karaman, Sertac},
	booktitle={ICRA},
	year={2018}
}
@article{ma2018self,
	title={Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera},
	author={Ma, Fangchang and Cavalheiro, Guilherme Venturelli and Karaman, Sertac},
	journal={arXiv preprint arXiv:1807.00275},
	year={2018}
}

Please create a new issue for code-related questions. Pull requests are welcome.