Awesome

Winning Solution in MIPI2022 Challenges on RGB+ToF Depth Completion

Requirements

Required

• pytorch
• numpy
• pillow
• opencv-python-headless
• scipy
• Matplotlib
• torch_ema

Optional

• tqdm
• tensorboardX

Pre-trained models

Download the pretrained models from Google Drive

Quickstart

Training

1. Step 1: download training data and fixed validation data from Google Drive and unzip them.

2. Step 2:

   • Train set: Record the path of the data pairs to a text file like this and assign the file location to the variable <font color="brown">'train_txt'</font> in <font color="brown">./utils/dataset.py</font>. Also, modify the data directory path in the member function <font color="brown">'self._load_png'</font>.
   • Val set: Processing is similar to the above.
   • Note that <font color="brown">'BeachApartmentInterior_My_ir'</font> scene's folder is removed from the training set, as it is partitioned into the fixed validation set.

3. Step 3:

   bash train.sh
   

Test

1. Step1:

   download the official test data and put it in <font color="blue">./Submit</font>

   download the pretrained model and put it in <font color="blue">./checkpoints</font>

2. Step2:

   cd ./Submit
   cp ../utils/define_model.py ./
   cp -R ../models ./
   bash test.sh 
   

3. Step 3: Check the results under the path <font color="brown">./Submit/results</font>

Citation

If you find our codes useful for your research, please consider citing our paper: (TBD)

[1] Dewang Hou, Yuanyuan Du, Kai Zhao, and Yang Zhao, "Learning an Efficient Multimodal Depth Completion Model", <i>1st MIPI: Mobile Intelligent Photography & Imaging workshop and challenge on RGB+ToF depth completion in conjunction with ECCV 2022. </i> [PDF] [arXiv]

@inproceedings{hou2023learning,
  title={Learning an Efficient Multimodal Depth Completion Model},
  author={Hou, Dewang and Du, Yuanyuan and Zhao, Kai and Zhao, Yang},
  booktitle={Computer Vision--ECCV 2022 Workshops: Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part V},
  pages={161--174},
  year={2023},
  organization={Springer}
}