Awesome
P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting
Created by Ziyi Wang*, Xumin Yu*, Yongming Rao*, Jie Zhou, Jiwen Lu
This repository is an official implementation of P2P (NeurIPS 2022 Spotlight).
<div align=left> <img src='https://github.com/wangzy22/P2P/blob/master/fig/p2p.gif' width=340 height=200> </div>P2P is a framework to leverage pre-trained image models for 3D analysis. We transform point clouds into colorful images with newly proposed geometry-preserved projection and geometry-aware coloring to adapt to pre-trained image models, whose weights are kept frozen during the end-to-end optimization of point cloud understanding tasks.
[arXiv][Project Page][Models]
Preparation
Installation Prerequisites
- Python 3.9
- CUDA 11.3
- PyTorch 1.11.1
- timm 0.5.4
- torch_scatter
- pointnet2_ops
- cv2, sklearn, yaml, h5py
conda create -n p2p python=3.9
conda activate p2p
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3
mkdir lib
cd lib
git clone https://github.com/erikwijmans/Pointnet2_PyTorch.git
cd Pointnet2_PyTorch
pip install pointnet2_ops_lib/.
cd ../..
pip install torch-scatter -f https://data.pyg.org/whl/torch-1.11.0+cu113.html
pip install timm==0.5.4
pip install opencv-python
pip install scikit-learn, h5py
conda install pyyaml, tqdm
Data Preparation
-
Download the processed ModelNet40 dataset from [Google Drive][Tsinghua Cloud][BaiDuYun](code:4u1e). Or you can download the offical ModelNet from here, and process it by yourself.
-
Download the official ScanObjectNN dataset from here.
-
The data is expected to be in the following file structure:
P2P/ |-- config/ |-- data/ |-- ModelNet40/ |-- modelnet40_shape_names.txt |-- modelnet_train.txt |-- modelnet_test.txt |-- modelnet40_train_8192pts_fps.dat |-- modelnet40_test_8192pts_fps.dat |-- ScanObjectNN/ |-- main_split/ |-- training_objectdataset_augmentedrot_scale75.h5 |-- test_objectdataset_augmentedrot_scale75.h5 |-- dataset/
Usage
Train
bash tool/train.sh EXP_NAME CONFIG_PATH DATASET
For example, to train P2P model with ConvNeXt-B-1k as base model on the ModelNet40 dataset:
bash tool/train.sh p2p_ConvNeXt-B-1k config/ModelNet40/p2p_ConvNeXt-B-1k.yaml ModelNet40
Test
bash tool/test.sh EXP_NAME CONFIG_PATH DATASET
For example, to test P2P model with ConvNeXt-B-1k as base model the ModelNet40 dataset:
bash tool/test.sh p2p_ConvNeXt-B-1k config/ModelNet40/p2p_ConvNeXt-B-1k.yaml ModelNet40
Reproduce
bash tool/reproduce.sh DATASET MODEL
For example, to reproduce results of P2P model with ConvNeXt-B-1k as base model on the ModelNet40 dataset with our provided checkpoint:
bash tool/reproduce.sh ModelNet40 ConvNeXt-B-1k
Results
Quantitative Results
We provide pretrained P2P models:
To train our P2P framework with HorNet-L-22k as the image model, please download the pre-trained weight from the official HorNet repo. Please organize the directory as the following structure:
P2P/
|-- pretrained/
|-- hornet_large_gf_in22k.pth
|-- reproduce/
|-- ckpt/
|-- ModelNet40/
|-- ConvNeXt-B-1k-ModelNet40.pth
|-- ScanObjectNN/
|-- ConvNeXt-B-1k-ScanObjectNN.pth
Visualization Results
Citation
If you find our work useful in your research, please consider citing:
@article{wang2022p2p,
title={P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting},
author={Wang, Ziyi and Yu, Xumin and Rao, Yongming and Zhou, Jie and Lu, Jiwen},
journal={arXiv preprint arXiv:2208.02812},
year={2022}
}
Acknowledgements
Our code is inspired by BPNet, Point-BERT.