Awesome
PlaneDepth
This is the official PyTorch implementation for the CVPR 2023 paper
<p align="center"> <img src="figures/pipeline.png" alt="pipeline of our method" width="1000" /> </p>PlaneDepth: Self-supervised Depth Estimation via Orthogonal Planes
π Setup
We recommend using anaconda to create the env and install the requirements by running:
conda create -n planedepth python=3.9.7
conda activate planedepth
conda install pytorch==1.10.0 torchvision==0.11.1 torchaudio==0.10.0 cudatoolkit=11.3.1 -c pytorch -c conda-forge
pip install -r requirements.txt
π KITTI training data
You can download the entire raw KITTI dataset by running:
wget -i splits/kitti_archives_to_download.txt -P kitti/
Then unzip with
cd kitti
unzip "*.zip"
cd ..
You can also place the KITTI dataset wherever you like and point towards it with the --data_path
flag during training and evaluation.
π Training
We provide the defult stereo training command of stage1 in train_ResNet.sh
.
To perform HRfinetune after stage1, update train_ResNet.sh
as:
CUDA_VISIBLE_DEVICES=0,1,2,3 OMP_NUM_THREADS=1 torchrun --nproc_per_node=4 train.py \
--png \
--model_name exp1_HR \ # modified
--use_denseaspp \
--use_mixture_loss \
--plane_residual \
--flip_right \
--learning_rate 2.5e-5 \ # modified
--num_epochs 1 \ # modified
--width 1280 \ # new
--height 384 \ # new
--no_crop \ # new
--load_weights_folder ./log/ResNet/exp1/last_models \ # new
--models_to_load depth encoder # new
To perform self-distillation after HRfinetune, update train_ResNet.sh
as:
CUDA_VISIBLE_DEVICES=0,1,2,3 OMP_NUM_THREADS=1 torchrun --nproc_per_node=4 train.py \
--png \
--model_name exp1_sd \ # modified
--use_denseaspp \
--use_mixture_loss \
--plane_residual \
--batch_size 4 \ # modified
--learning_rate 2e-5 \ # modified
--num_epochs 10 \ # modified
--milestones 5 \ # modified
--width 1280 \
--height 384 \
--no_crop \
--load_weights_folder ./log/ResNet/exp1_HR/last_models \ # modified
--models_to_load depth encoder \
--self_distillation 1. # new
Monocular training:
Please adjust the following flags:
--warp_type homography_warp
--split eigen_zhou
--novel_frame_ids 1 -1
--automask
(optional) --no_stereo
(optional) --use_colmap
Other training options
Look at options.py
to see other options.
π Evaluation
We provide the defult evaluation command in eval.sh
. Please refer to your training settings to modify it.
Prepare Eigen raw ground truth
You may need to export the ground truth depth before evaluation on Eigen raw split. Please run:
python splits/eigen_raw/export_gt_depth.py --data_path ./kitti
Prepare Eigen improved ground truth
To perform Eigen improved evaluation, you need to download the Eigen improved dataset (14GB) and unzip it by running:
unzip data_depth_annotated.zip -d kitti_depth
You can also place it wherever you like and point towards it with the --improved_path flag during export:
python splits/eigen_improved/prepare_groundtruth.py --improved_path ./kitti_depth
π Pretrained model
Model | Abs Rel | A1 |
---|---|---|
stage1 | 0.089 | 0.900 |
HRfinetune | 0.086 | 0.906 |
self-distillation | 0.085 | 0.910 |
π Known issues
- When using the flag --use_mixture_loss in the
train.py
, users may encounter the error message "CUDNN_STATUS_NOT_INITIALIZED". This issue may be resolved by reducing the batch_size. Issue_4
π TODO list
- The ground truth depth during training is wrong because of cropping, which will influence the training log in tensorboard.
π Acknowledgements
We thank Monodepth2 and FalNet for their outstanding methods and codes.
π Citation
If you find our paper or code useful, please cite
@inproceedings{wang2023planedepth,
author = {Wang, Ruoyu and Yu, Zehao and Gao, Shenghua},
title = {PlaneDepth: Self-Supervised Depth Estimation via Orthogonal Planes},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2023},
pages = {21425-21434}
}
π Contact us
If you have any questions, don't hesitate to contact us at wangry3@shanghaitech.edu.cn or open an issue. Let's discuss and create more sparkling⨠works!