Home

Awesome

Awesome Depth Completion

Table of Contents

About Sparse-to-Dense Depth Completion <a name="about-sparse-to-dense-depth-completion"></a>

In the sparse-to-dense depth completion problem, one wants to infer the dense depth map of a 3-D scene given an RGB image and its corresponding sparse reconstruction in the form of a sparse depth map obtained either from computational methods such as SfM (Strcuture-from-Motion) or active sensors such as lidar or structured light sensors.

Example 1: VOID dataset (indoor VIO)

Input RGB imageSparse point cloudOutput point cloud from KBNet
<img src="figures/void_teaser_image_306.png" width="200"><img src="figures/void_teaser_sparse_point_cloud_306.gif" width="200"><img src="figures/void_teaser_kbnet_output_306.gif" width="200">

Example 2: KITTI dataset (outdoor lidar)

Input RGB imageOutput point cloud from ScaffNet
<img src="figures/kitti_teaser_image.png" width="400"><img src="figures/kitti_teaser_scaffnet_output.gif" width="400">

Current State of Depth Completion Methods <a name="current-state-of-depth-completion"></a>

Here we compile both unsupervised/self-supervised (monocular and stereo) and supervised methods published in recent conferences and journals on the VOID (Wong et. al., 2020) and KITTI (Uhrig et. al., 2017) depth completion benchmarks. Our ranking considers all four metrics rather than just RMSE.

Quick Links

Unsupervised VOID Depth Completion Benchmark <a name="unsupervised-void-benchmark"></a>

PaperPublicationCodeMAERMSEiMAEiRMSE
Monitored Distillation for Positive Congruent Depth CompletionECCV 2022PyTorch36.4287.7819.1843.83
Unsupervised Depth Completion with Calibrated Backprojection LayersICCV 2021PyTorch39.8095.8621.1649.72
Learning Topology from Synthetic Data for Unsupervised Depth CompletionRA-L & ICRA 2021Tensorflow/PyTorch60.68122.0135.2467.34
Unsupervised Depth Completion from Visual Inertial OdometryRA-L & ICRA 2020Tensorflow/PyTorch85.05169.7948.92104.02
Dense depth posterior (ddp) from single image and sparse rangeCVPR 2019Tensorflow151.86222.3674.59112.36
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular CameraICRA 2019PyTorch178.85243.8480.12107.69

Supervised VOID Depth Completion Benchmark <a name="supervised-void-benchmark"></a>

PaperPublicationCodeMAERMSEiMAEiRMSE
CostDCNet: Cost Volume based Depth Completion for a Single RGB-D ImageECCV 2022PyTorch25.8476.2812.1932.13
Non-Local Spatial Propagation Network for Depth CompletionECCV 2020PyTorch26.7479.1212.7033.88
Monitored Distillation for Positive Congruent Depth CompletionECCV 2022PyTorch29.6779.7814.8437.88
PENet: Towards Precise and Efficient Image Guided Depth Completion (PENet)ICRA 2021PyTorch34.6182.0118.8940.36
A Multi-Scale Guided Cascade Hourglass Network for Depth CompletionWACV 2020PyTorch43.57109.9423.4452.09
PENet: Towards Precise and Efficient Image Guided Depth Completion (ENet)ICRA 2021PyTorch46.9094.3526.7852.58
Scanline Resolution-Invariant Depth Completion Using a Single Image and Sparse LiDAR Point CloudRA-L & IROS 2021N/A59.40181.4219.3746.56

Unsupervised KITTI Depth Completion Benchmark <a name="unsupervised-kitti-benchmark"></a>

PaperPublicationCodeMAERMSEiMAEiRMSE
Unsupervised Depth Completion with Calibrated Backprojection LayersICCV 2021PyTorch256.761069.471.022.95
Learning Topology from Synthetic Data for Unsupervised Depth CompletionRA-L & ICRA 2021Tensorflow280.761121.931.153.30
Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor DataACCV 2020PyTorch280.421095.261.193.53
Unsupervised Depth Completion from Visual Inertial OdometryRA-L & ICRA 2020Tensorflow299.411169.971.203.56
A Surface Geometry Model for LiDAR Depth CompletionRA-L & ICRA 2021Tensorflow298.31239.841.213.76
Dense depth posterior (ddp) from single image and sparse rangeCVPR 2019Tensorflow343.461263.191.323.58
DFuseNet: Deep Fusion of RGB and Sparse Depth Information for Image Guided Dense Depth CompletionITSC 2019PyTorch429.931206.661.793.62
In Defense of Classical Image Processing: Fast Depth Completion on the CPUCRV 2018Python302.601288.461.293.78
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular CameraICRA 2019PyTorch350.321299.851.574.07
Semantically Guided Depth UpsamplingGCPR 2016N/A605.472312.572.057.38

Supervised KITTI Depth Completion Benchmark <a name="supervised-kitti-benchmark"></a>

PaperPublicationCodeMAERMSEiMAEiRMSE
Dynamic Spatial Propagation Network for Depth CompletionAAAI 2022N/A192.71709.120.821.88
Non-Local Spatial Propagation Network for Depth CompletionECCV 2020PyTorch199.5741.680.841.99
CompletionFormer: Depth Completion with Convolutions and Vision TransformersCVPR 2023PyTorch203.45708.870.882.01
SemAttNet: Towards Attention-based Semantic Aware Guided Depth CompletionIEEE Access 2022PyTorch205.49709.410.902.03
RigNet: Repetitive Image Guided Network for Depth CompletionECCV 2022N/A203.25712.660.902.08
MFF-Net: Towards Efficient Monocular Depth Completion with Multi-modal Feature FusionRAL 2023N/A208.11719.850.942.21
CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth CompletionAAAI 2020N/A209.28743.690.902.07
Dense depth posterior (ddp) from single image and sparse rangeCVPR 2019Tensorflow203.96832.940.852.10
Adaptive context-aware multi-modal network for depth completionTIP 2021PyTorch206.80732.990.902.08
PENet: Towards Precise and Efficient Image Guided Depth CompletionICRA 2021PyTorch210.55730.080.942.17
Monitored Distillation for Positive Congruent Depth CompletionECCV 2022PyTorch218.60785.060.922.11
MDANet: Multi-Modal Deep Aggregation Network for Depth CompletionICRA 2021PyTorch214.99738.230.992.12
A Cascade Dense Connection Fusion Network for Depth CompletionBMVC 2022N/A216.05738.260.992.18
FCFR-Net: Feature Fusion based Coarse- to-Fine Residual Learning for Depth CompletionAAAI 2021N/A217.15735.810.982.20
Learning Guided Convolutional Network for Depth CompletionTIP 2020PyTorch218.83736.240.992.25
DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion NetworkICRA 2021N/A214.13755.410.962.25
A Multi-Scale Guided Cascade Hourglass Network for Depth CompletionWACV 2020PyTorch220.41762.190.982.30
Sparse and noisy LiDAR completion with RGB guidance and uncertaintyMVA 2019PyTorch215.02772.870.932.19
A Multi-Scale Guided Cascade Hourglass Network for Depth CompletionWACV 2020N/A220.41762.190.982.30
Learning Joint 2D-3D Representations for Depth CompletionICCV 2019N/A221.19752.881.142.34
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color ImageCVPR 2019PyTorch226.50758.381.152.56
Depth Completion from Sparse LiDAR Data with Depth-Normal ConstraintsICCV 2019N/A235.17777.051.132.42
Scanline Resolution-Invariant Depth Completion Using a Single Image and Sparse LiDAR Point CloudRA-L & IROS 2021N/A233.34809.091.062.57
Confidence propagation through cnns for guided sparse depth regressionPAMI 2019PyTorch233.26829.981.032.60
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular CameraICRA 2019PyTorch249.95814.731.212.80
Uncertainty-Aware CNNs for Depth Completion: Uncertainty from Beginning to EndCVPR 2020PyTorch251.77960.051.053.37
Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation3DV 2019N/A234.81917.640.952.17
Depth coefficients for depth completionCVPR 2019N/A252.21988.381.132.87
Depth estimation via affinity learned with convolutional spatial propagation networkECCV 2018N/A279.461019.641.152.93
Learning morphological operators for depth completionACIVS 2019N/A310.491045.451.573.84
Sparsity Invariant CNNs3DV 2017Tensorflow416.141419.751.293.25
Deep Convolutional Compressed Sensing for LiDAR Depth CompletionACCV 2018Tensorflow439.481325.373.1959.39