Home

Awesome

Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models. (ECCV 2022)

By Chenfeng Xu*, Shijia Yang*, Tomer Galanti, Bichen Wu, Bohan Zhai, Xiangyu Yue, Wei Zhan, Peter Vajda, Kurt Keutzer, Masayoshi Tomizuka.

Our paper explores the potential of transferring 2D model architectures and weights to understand 3D point-clouds, by empirically investigating the feasibility of the transfer, the benefits of the transfer, and shedding light on why the transfer works. We discover that we can indeed use the same architecture and pretrained weights of a neural net model to understand both images and point-clouds. Specifically, we transfer the image-pretrained model to a point-cloud model by copying or inflating the weights. We find that finetuning the transformed image-pretrained models (FIP) with minimal efforts -- only on input, output, and normalization layers -- can achieve competitive performance on 3D point-cloud classification, beating a wide range of point-cloud models that adopt task-specific architectures and use a variety of tricks. When finetuning the whole model, the performance improves even further. Meanwhile, FIP improves data efficiency, reaching up to 10.0 top-1 accuracy percent on few-shot classification. It also speeds up the training of point-cloud models by up to 11.1x for a target accuracy (e.g., 90 % accuracy). Lastly, we provide an explanation of the image to point-cloud transfer from the aspect of neural collapse. The paper can be found at Arxiv.

This repository contains a Pytorch implementation of Image2Point. The framework of our work can be found below:

<p align="center"> <img src="./pic.png"/ width="900"> </p>

If you find it helpful, please consider cite it as

Citation

@inproceedings{xu2022image2point,
  title={Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models},
  author={Xu, Chenfeng and Yang, Shijia and Galanti, Tomer and Wu, Bichen and Yue, Xiangyu and Zhai, Bohan and Zhan, Wei and Vajda, Peter and Keutzer, Kurt and Tomizuka, Masayoshi},
  booktitle={European Conference on Computer Vision},
  pages={638--656},
  year={2022},
  organization={Springer}
}

License

Image2Point is released under the BSD license (See LICENSE for details).

Installation

The instructions are tested on Ubuntu 16.04 with python 3.7.5 and Pytorch 1.7.0 with GPU support.

git clone https://github.com/chenfengxu714/image2point.git

conda create --name i2p python=3.7.5

conda activate i2p

conda install pytorch=1.7 torchvision torchaudio cudatoolkit=10.2 -c pytorch

pip install -r requirements.txt

pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git@v1.2.0

Pretrained Models

Coming soon.

Training

General command:

torchpack dist-run -np [num_of_gpus] python train.py configs/[dataset]/[config_name].yaml --run-dir [output_dir_name]

To train a model from scratch, use the following [dataset]/[config_name].yaml:

To train a model using pretrained 2D ConvNets, use the following [dataset]/[config_name].yaml:

To reproduce the semi-supervised learner experiment, first run few-shot setting on pretrained 2D ConvNets, then use the following:

--run-dir is optional. Output directory name will be automatically generated if not set.

Config files follow the training recipe are reported in the paper. The following fields change as experiment varies:

We can monitor the training process using tensorboard.

tensorboard --logdir tensorboard/[output_dir_name]

Evaluation

Coming soon.

Credits

We referred to SPVNAS (Paper, Code) during our development. We thank the authors of SPVNAS for open-sourcing their code.