Awesome

[ECCV2024] Temporally Consistent Stereo Matching

News

• 2024.07.06: The code will be uploaded in a few weeks.
• 2024.07.17: The paper and supplementary materials are available now.
• 2024.07.22: The code is available now. Video demos are coming soon.
• 2024.09.26: Video demos are available now.

Pipeline

Visualization

• TartanAir:

• KITTI:

Installation

Our code is based on CUDA 11.7 and PyTorch 2.0.1. We recommend using Anaconda to create a new environment:

conda create -n tcstereo python=3.8
conda activate tcstereo

Then, install the dependencies:

pip install -r requirements.txt

Dataset

We use the TartanAir, SceneFlow, and Raw data of KITTI datasets in our experiments. Please download the datasets and organize them as follows:

datasets
├── TartanAir
│   ├── abandonedfactory
│   │   ├── abandonedfactory
│   │   │   ├── Easy
│   │   │   │   ├── P000
│   │   │   │   │   ├── depth_left
│   │   │   │   │   ├── image_left
│   │   │   │   │   ├── image_right
│   │   │   │   │   ├── pose_left.txt
│   │   │   │   ├── ...
│   │   │   ├── Hard
│   │   │   │   ├── ...
│   ├── ...
├── Driving
│   ├── frames_cleanpass
│   ├── frames_finalpass
│   ├── disparity
│   ├── pose
├── FlyingThings3D
│   ├── frames_cleanpass
│   ├── frames_finalpass
│   ├── disparity
│   ├── pose
├── Monkaa
│   ├── frames_cleanpass
│   ├── frames_finalpass
│   ├── disparity
│   ├── pose
├── kitti_raw  # training sequences & pesudo labels, the pose files are generated by core/utils/preprocess_kitti_raw_pose.py
│   ├── 2011_09_26
│   ├── ...
├── KITTI  # testing sequences
│   ├── kitti_seq
│   │   ├── kitti2015_testings
│   │   │   ├──000000
│   │   │   ├── ...

Checkpoints

We provide the trained models on TartanAir, SceneFlow, and KITTI_raw datasets. Please download the checkpoints from Dropbox.

Evaluation

Before evaluation, please download the checkpoints and put them in the ./checkpoints directory.

You can evaluate the pre-trained models on TartanAir, SceneFlow, and KITTI_raw datasets by running the following scripts:

bash tartanair_evaluate.sh

bash sceneflow_evaluate.sh

bash submit_kitti.sh

Training

For TartanAir dataset, you can train the model by running the following script:

bash tartanair_ddp_train.sh

For SceneFlow dataset, you can train the model by running the following script:

bash sceneflow_ddp_train.sh

For KITTI_raw dataset, you can train the model based on the pre-trained model on TartanAir dataset by running the following script:

bash KITTI_ddp_train.sh

Acknowledgement

Our code is based on RAFT-Stereo, TemporalStereo, IGEV-Stereo and HITNet. We thank the authors for their great works.

Citation

If you find our work useful in your research, please consider citing:

@article{zeng2024temporally,
  title={Temporally Consistent Stereo Matching},
  author={Zeng, Jiaxi and Yao, Chengtang and Wu, Yuwei and Jia, Yunde},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2024}
}