Awesome

UMIFormer

This repository contains the source code for the paper UMIFormer: Mining the Correlations between Similar Tokens for Multi-View 3D Reconstruction.

Performance

* The results in this row are derived from models that train individually for the various number of input views.

Demo

Cite this work

@article{zhu2023umiformer,
  title={UMIFormer: Mining the Correlations between Similar Tokens for Multi-View 3D Reconstruction},
  author={Zhu, Zhenwei and Yang, Liying and Li, Ning and Jiang, Chaohao and Liang, Yanyan},
  journal={arXiv preprint arXiv:2302.13987},
  year={2023}
}

Datasets

We use the ShapeNet in our experiments, which are available below:

• ShapeNet rendering images: http://cvgl.stanford.edu/data2/ShapeNetRendering.tgz
• ShapeNet voxelized models: http://cvgl.stanford.edu/data2/ShapeNetVox32.tgz

Pretrained Models

The pretrained models on ShapeNet are available as follows:

• UMIFormer
• UMIFormer+

Please download them and put into ./pths/

Prerequisites

Clone the Code Repository

git clone https://github.com/GaryZhu1996/UMIFormer

Install Python Dependencies

cd UMIFormer
conda env create -f environment.yml

Modify the path of datasets

Modify __C.DATASETS.SHAPENET.RENDERING_PATH and __C.DATASETS.SHAPENET.VOXEL_PATH in config.py to the correct path of ShapeNet dataset.

3D Reconstruction Model

For training, please use the following command:

CUDA_VISIBLE_DEVICES=gpu_ids python -m torch.distributed.launch --nproc_per_node=num_of_gpu runner.py

For testing, please follow the steps below:

1. Update the setting of __C.CONST.WEIGHTS in config.py as the path of the reconstruction model;
2. Run the following command to evaluate the performance of the model when facing the number of input views defined by __C.CONST.N_VIEWS_RENDERING in config.py:

CUDA_VISIBLE_DEVICES=gpu_ids python -m torch.distributed.launch --nproc_per_node=num_of_gpu runner.py --test

3. Run the following command to evaluate the performance of the model when facing various numbers of input views mentioned in the paper:

CUDA_VISIBLE_DEVICES=gpu_ids python -m torch.distributed.launch --nproc_per_node=num_of_gpu runner.py --batch_test

Our Other Works on Multi-View 3D Reconstruction

@article{zhu2023garnet,
  title={GARNet: Global-Aware Multi-View 3D Reconstruction Network and the Cost-Performance Tradeoff},
  author={Zhu, Zhenwei and Yang, Liying and Lin, Xuxin and Yang, Lin and Liang, Yanyan},
  journal={Pattern Recognition},
  pages={109674},
  year={2023},
  publisher={Elsevier}
}

@InProceedings{Yang_2023_ICCV,
    author    = {Yang, Liying and Zhu, Zhenwei and Lin, Xuxin and Nong, Jian and Liang, Yanyan},
    title     = {Long-Range Grouping Transformer for Multi-View 3D Reconstruction},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {18257-18267}
}

License

This project is open sourced under MIT license.