Home

Awesome

IDPT: Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models

PWC PWC PWC

This repository provides the official implementation of Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models at ICCV 2023.

šŸ“Ø News

1. Introduction

We first explore prompt tuning for pre-trained point cloud models and propose a novel Instance-aware Dynamic Prompt Tuning (IDPT) strategy to enhance the model robustness against distributional diversity (caused by various noises) in real-world point clouds. IDPT generally utilizes a lightweight prompt generation module to perceive the semantic prior features and generate instance-aware prompt tokens for the pre-trained point cloud model. Compared with the common (static) prompt tuning strategies like Visual Prompt Tuning (VPT), IDPT shows notable improvement in downstream adaptation. IDPT is also competitive with full fine-tuning while requiring only ~7% of the trainable parameters.

img.png

In the following, we will guide you how to use this repository step by step. šŸ¤—

2. Preparation

git clone git@github.com:zyh16143998882/ICCV23-IDPT.git
cd ICCV23-IDPT/

2.1 Requirements

conda create -y -n idpt python=3.7
conda activate idpt
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt

# Chamfer Distance & emd
cd ./extensions/chamfer_dist
python setup.py install --user
cd ./extensions/emd
python setup.py install --user

# PointNet++
pip install "git+https://github.com/erikwijmans/Pointnet2_PyTorch.git#egg=pointnet2_ops&subdirectory=pointnet2_ops_lib"

# GPU kNN
pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl
pip install torch-scatter

2.2 Download the point cloud datasets and organize them properly

Before running the code, we need to make sure that everything needed is ready. First, the working directory is expected to be organized as below:

<details><summary>click to expand šŸ‘ˆ</summary>
ICCV23-IDPT/
ā”œā”€ā”€ cfgs/
ā”œā”€ā”€ data/
ā”‚   ā”œā”€ā”€ ModelNet/ # ModelNet40
ā”‚   ā”‚   ā””ā”€ā”€ modelnet40_normal_resampled/
ā”‚   ā”‚       ā”œā”€ā”€ modelnet40_shape_names.txt
ā”‚   ā”‚       ā”œā”€ā”€ modelnet40_train.txt
ā”‚   ā”‚       ā”œā”€ā”€ modelnet40_test.txt
ā”‚   ā”‚       ā”œā”€ā”€ modelnet40_train_8192pts_fps.dat
ā”‚   ā”‚       ā””ā”€ā”€ modelnet40_test_8192pts_fps.dat
ā”‚   ā”œā”€ā”€ ModelNetFewshot/ # ModelNet Few-shot
ā”‚   ā”‚   ā”œā”€ā”€ 5way10shot/
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ 0.pkl
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ ...
ā”‚   ā”‚   ā”‚   ā””ā”€ā”€ 9.pkl
ā”‚   ā”‚   ā”œā”€ā”€ 5way20shot/
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ ...
ā”‚   ā”‚   ā”‚   ...
ā”‚   ā”‚   ā”œā”€ā”€ 10way10shot/
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ ...
ā”‚   ā”‚   ā”‚   ...
ā”‚   ā”‚   ā””ā”€ā”€ 10way20shot/
ā”‚   ā”‚       ā”œā”€ā”€ ...
ā”‚   ā”‚       ...
ā”‚   ā”œā”€ā”€ ScanObjectNN/ # ScanObjectNN
ā”‚   ā”‚   ā”œā”€ā”€ main_split/
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ training_objectdataset_augmentedrot_scale75.h5
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ test_objectdataset_augmentedrot_scale75.h5
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ training_objectdataset.h5
ā”‚   ā”‚   ā”‚   ā””ā”€ā”€ test_objectdataset.h5
ā”‚   ā”‚   ā””ā”€ā”€ main_split_nobg/
ā”‚   ā”‚       ā”œā”€ā”€ training_objectdataset.h5
ā”‚   ā”‚       ā””ā”€ā”€ test_objectdataset.h5
ā”‚   ā”œā”€ā”€ ShapeNet55-34/ # ShapeNet55/34
ā”‚   ā”‚   ā”œā”€ā”€ shapenet_pc/
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ 02691156-1a04e3eab45ca15dd86060f189eb133.npy
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ 02691156-1a6ad7a24bb89733f412783097373bdc.npy
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ ...
ā”‚   ā”‚   ā”‚   ...
ā”‚   ā”‚   ā””ā”€ā”€ ShapeNet-55/
ā”‚   ā”‚       ā”œā”€ā”€ train.txt
ā”‚   ā”‚       ā””ā”€ā”€ test.txt
ā”‚   ā””ā”€ā”€ shapenetcore_partanno_segmentation_benchmark_v0_normal/ # ShapeNetPart
ā”‚       ā”œā”€ā”€ 02691156/
ā”‚       ā”‚   ā”œā”€ā”€ 1a04e3eab45ca15dd86060f189eb133.txt
ā”‚       ā”‚   ā”œā”€ā”€ ...
ā”‚       ā”‚   ...
ā”‚       ā”‚ā”€ā”€ ...
ā”‚       ā”‚ā”€ā”€ train_test_split/
ā”‚       ā””ā”€ā”€ synsetoffset2category.txt
ā”œā”€ā”€ datasets/
ā”œā”€ā”€ ...
...
</details>

Here we have also collected the download links of required datasets for you:

3. Pre-train a point cloud model (e.g. Point-MAE)

To pre-train Point-MAE on ShapeNet training set, you can run the following command:

# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/pretrain.yaml --exp_name <output_file_name>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/pretrain.yaml --exp_name pretrain_pointmae

If you want to try other models or change pre-training configuration, e.g., mask ratios, just create a new configuration file and pass its path to --config.

For a quick start, we also have provided the pre-trained checkpoint of Point-MAE [link].

4. Tune pre-trained point cloud models on downstream tasks

We take VPT and IDPT as two showcases of prompt tuning for pre-trained point cloud models. Executable commands of different downstream tasks are listed below.

4.1 Object Classification

4.1.1 ModelNet40

<details><summary>VPT-Deep (click to expand šŸ‘ˆ)</summary>
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_modelnet_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name modelnet_vpt

# further enable voting mechanism
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet.yaml --test --vote --exp_name modelnet_vpt_vote --ckpts ./experiments/finetune_modelnet/cfgs/modelnet_vpt/ckpt-best.pth
</details> <details><summary>IDPT (click to expand šŸ‘ˆ)</summary>
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_modelnet_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name modelnet_idpt

# further enable voting mechanism
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet.yaml --test --vote --exp_name modelnet_idpt_vote --ckpts ./experiments/finetune_modelnet/cfgs/modelnet_idpt/ckpt-best.pth
</details>

4.1.2 ScanObjectNN (OBJ-BG)

<details><summary>VPT-Deep (click to expand šŸ‘ˆ)</summary>
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objbg_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objbg_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name bg_vpt
</details> <details><summary>IDPT (click to expand šŸ‘ˆ)</summary>
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objbg_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objbg_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name bg_idpt
</details>

4.1.3 ScanObjectNN (OBJ-ONLY)

<details><summary>VPT-Deep (click to expand šŸ‘ˆ)</summary>
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objonly_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objonly_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name only_vpt
</details> <details><summary>IDPT (click to expand šŸ‘ˆ)</summary>
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objonly_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objonly_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name only_idpt
</details>

4.1.4 ScanObjectNN (PB-T50-RS)

<details><summary>VPT-Deep (click to expand šŸ‘ˆ)</summary>
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_hardest_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_hardest_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name hard_vpt
</details> <details><summary>IDPT (click to expand šŸ‘ˆ)</summary>
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_hardest_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_hardest_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name hard_idpt
</details>

4.2 Few-shot Learning on ModelNet Few-shot

<details><summary>VPT-Deep (click to expand šŸ‘ˆ)</summary>
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/fewshot_vpt.yaml --finetune_model --ckpts <path/to/pre-trained/model> --exp_name <output_file_name> --way <5 or 10> --shot <10 or 20> --fold <0-9>
for WAY in 5 10
do
  for SHOT in 10 20
  do
    for FOLD in $(seq 0 9)
    do
	  CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/fewshot_vpt.yaml --finetune_model --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --exp_name fewshot_vpt --way ${WAY} --shot ${SHOT} --fold ${FOLD}
    done
  done
done
</details> <details><summary>IDPT (click to expand šŸ‘ˆ)</summary>
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/fewshot_idpt.yaml --finetune_model --ckpts <path/to/pre-trained/model> --exp_name <output_file_name> --way <5 or 10> --shot <10 or 20> --fold <0-9>
for WAY in 5 10
do
  for SHOT in 10 20
  do
    for FOLD in $(seq 0 9)
    do
	  CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/fewshot_idpt.yaml --finetune_model --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --exp_name fewshot_idpt --way ${WAY} --shot ${SHOT} --fold ${FOLD}
    done
  done
done
</details>

4.3 Part Segmentation on ShapeNet-Part

<details><summary>IDPT (click to expand šŸ‘ˆ)</summary>
cd segmentation

# python main.py --model prompt_pt3 --optimizer_part only_new --ckpts <path/to/pre-trained/model> --root path/to/data --learning_rate 0.0002 --epoch 300
CUDA_VISIBLE_DEVICES=0 python main.py --model prompt_pt3 --optimizer_part only_new --ckpts ../checkpoint/pretrain/mae/ckpt-last.pth --root ../data/shapenetcore_partanno_segmentation_benchmark_v0_normal/ --log_dir seg_idpt --learning_rate 0.0002 --epoch 300
</details>

5. Validate with checkpoints

For reproducibility, logs and checkpoints of tuned models via IDPT can be found in the table below.

TaskDatasetTrainable ParameterslogAcc.Checkpoints Download
Pre-trainingShapeNet1.7M-N.A.Point-MAE
ClassificationScanObjectNN1.7Mfinetune_scan_objbg.log93.63%OBJ-BG
ClassificationScanObjectNN1.7Mfinetune_scan_objonly.log93.12%OBJ-ONLY
ClassificationScanObjectNN1.7Mfinetune_scan_hardest.log88.51%PB-T50-RS
ClassificationModelNet401.7Mfinetune_modelnet.log93.3%ModelNet-1k
ClassificationModelNet40 (vote)-finetune_modelnet_vote.log94.4%-
TaskDatasetlog5w10s (%)5w20s (%)10w10s (%)10w20s (%)
Few-shot learningModelNet40fewshot_logs97.3 Ā± 2.197.9 Ā± 1.192.8 Ā± 4.195.4 Ā± 2.9
<!-- šŸ’”***Notes***: For classification downstream tasks, we randomly select 5 seeds to obtain the best checkpoint. -->

The evaluation commands with checkpoints should be in the following format:

CUDA_VISIBLE_DEVICES=<GPU> python main.py --test --config <yaml_file_name> --exp_name <output_file_name> --ckpts <path/to/ckpt>
<details><summary>For example, click to expand šŸ‘ˆ</summary>
# object classification on ScanObjectNN (PB-T50-RS)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_hardest_idpt.yaml --ckpts ./checkpoint/hardest/ckpt-best.pth --test --exp_name hard_test

# object classification on ScanObjectNN (OBJ-BG)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objbg_idpt.yaml --ckpts ./checkpoint/bg/ckpt-best.pth --test --exp_name bg_test

# object classification on ScanObjectNN (OBJ-ONLY)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objonly_idpt.yaml --ckpts ./checkpoint/only/ckpt-best.pth --test --exp_name only_test

# object classification on ModelNet40 (w/o voting)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_idpt.yaml --ckpts ./checkpoint/modelnet40/ckpt-best.pth --test --exp_name model_test

# object classification on ModelNet40 (w/ voting)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_idpt.yaml --test --vote --exp_name modelnet_idpt_vote --ckpts ./checkpoint/modelnet40/ckpt-best.pth

# few-show learning on ModelNet40
python parse_test_res.py ./experiments/all/fewshot --multi-exp --few-shot
</details>

6. t-SNE visualization

We used t-SNE to visualize the results obtained by IDPT on the ModelNet40 and ScanObjectNN test sets. The execution commands are as follows:

CUDA_VISIBLE_DEVICES=<GPU> python main.py --tsne --config <yaml_file_name> --exp_name <output_file_name> --ckpts <path/to/ckpt>
<details><summary>For example, click to expand šŸ‘ˆ</summary>
# t-SNE on ScanObjectNN (PB-T50-RS)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_scan_hardest_idpt_tsne.yaml --ckpts ./checkpoint/hardest/ckpt-best.pth --tsne --exp_name hard_test_tsne

# object classification on ScanObjectNN (OBJ-BG)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_scan_objbg_idpt_tsne.yaml --ckpts ./checkpoint/bg/ckpt-best.pth --tsne --exp_name bg_test_tsne

# object classification on ScanObjectNN (OBJ-ONLY)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_scan_objonly_idpt_tsne.yaml --ckpts ./checkpoint/only/ckpt-best.pth --tsne --exp_name only_test_tsne

# object classification on ModelNet40 
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_modelnet_idpt_tsne.yaml --ckpts ./checkpoint/modelnet40/ckpt-best.pth --tsne --exp_name model_test_tsne
</details>

You can also modify the code yourself to implement a customized t-SNE visualization. šŸ¤—

7. Bibliography

If you find this code useful or use the toolkit in your work, please consider citing:

@inproceedings{zha2023_IDPT,
  title={Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models},
  author={Zha, Yaohua and Wang, Jinpeng and Dai, Tao and Chen, Bin and Wang, Zhi and Xia, Shu-Tao},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2023}
}

8. Acknowledgements

Our codes are built upon Point-MAE, Point-BERT, ACT, DGCNN and VPT. Thanks for their efforts.

9. Contact

If you have any question, you can raise an issue or email Yaohua Zha (chayh21@mails.tsinghua.edu.cn) and Jinpeng Wang (wjp20@mails.tsinghua.edu.cn). We will reply you soon.