Awesome
IDPT: Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models
This repository provides the official implementation of Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models at ICCV 2023.
šØ News
- [2023.12.10] š„ Our paper Point-FEMAE (github) has been accepted by AAAI 2024! šš Many thanks to all the collaborators and anonymous reviewers! š¤
- [2023.07.18] š„ Release the code and instructions. š¤
- [2023.07.14] š„ Our paper IDPT has been accepted by ICCV 2023! šš Many thanks to all the collaborators and anonymous reviewers! š„°
1. Introduction
We first explore prompt tuning for pre-trained point cloud models and propose a novel Instance-aware Dynamic Prompt Tuning (IDPT) strategy to enhance the model robustness against distributional diversity (caused by various noises) in real-world point clouds. IDPT generally utilizes a lightweight prompt generation module to perceive the semantic prior features and generate instance-aware prompt tokens for the pre-trained point cloud model. Compared with the common (static) prompt tuning strategies like Visual Prompt Tuning (VPT), IDPT shows notable improvement in downstream adaptation. IDPT is also competitive with full fine-tuning while requiring only ~7% of the trainable parameters.
In the following, we will guide you how to use this repository step by step. š¤
2. Preparation
git clone git@github.com:zyh16143998882/ICCV23-IDPT.git
cd ICCV23-IDPT/
2.1 Requirements
- gcc >= 4.9
- cuda >= 9.0
- python >= 3.7
- pytorch >= 1.7.0 < 1.11.0
- anaconda
- torchvision
conda create -y -n idpt python=3.7
conda activate idpt
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt
# Chamfer Distance & emd
cd ./extensions/chamfer_dist
python setup.py install --user
cd ./extensions/emd
python setup.py install --user
# PointNet++
pip install "git+https://github.com/erikwijmans/Pointnet2_PyTorch.git#egg=pointnet2_ops&subdirectory=pointnet2_ops_lib"
# GPU kNN
pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl
pip install torch-scatter
2.2 Download the point cloud datasets and organize them properly
Before running the code, we need to make sure that everything needed is ready. First, the working directory is expected to be organized as below:
<details><summary>click to expand š</summary>ICCV23-IDPT/
āāā cfgs/
āāā data/
ā āāā ModelNet/ # ModelNet40
ā ā āāā modelnet40_normal_resampled/
ā ā āāā modelnet40_shape_names.txt
ā ā āāā modelnet40_train.txt
ā ā āāā modelnet40_test.txt
ā ā āāā modelnet40_train_8192pts_fps.dat
ā ā āāā modelnet40_test_8192pts_fps.dat
ā āāā ModelNetFewshot/ # ModelNet Few-shot
ā ā āāā 5way10shot/
ā ā ā āāā 0.pkl
ā ā ā āāā ...
ā ā ā āāā 9.pkl
ā ā āāā 5way20shot/
ā ā ā āāā ...
ā ā ā ...
ā ā āāā 10way10shot/
ā ā ā āāā ...
ā ā ā ...
ā ā āāā 10way20shot/
ā ā āāā ...
ā ā ...
ā āāā ScanObjectNN/ # ScanObjectNN
ā ā āāā main_split/
ā ā ā āāā training_objectdataset_augmentedrot_scale75.h5
ā ā ā āāā test_objectdataset_augmentedrot_scale75.h5
ā ā ā āāā training_objectdataset.h5
ā ā ā āāā test_objectdataset.h5
ā ā āāā main_split_nobg/
ā ā āāā training_objectdataset.h5
ā ā āāā test_objectdataset.h5
ā āāā ShapeNet55-34/ # ShapeNet55/34
ā ā āāā shapenet_pc/
ā ā ā āāā 02691156-1a04e3eab45ca15dd86060f189eb133.npy
ā ā ā āāā 02691156-1a6ad7a24bb89733f412783097373bdc.npy
ā ā ā āāā ...
ā ā ā ...
ā ā āāā ShapeNet-55/
ā ā āāā train.txt
ā ā āāā test.txt
ā āāā shapenetcore_partanno_segmentation_benchmark_v0_normal/ # ShapeNetPart
ā āāā 02691156/
ā ā āāā 1a04e3eab45ca15dd86060f189eb133.txt
ā ā āāā ...
ā ā ...
ā āāā ...
ā āāā train_test_split/
ā āāā synsetoffset2category.txt
āāā datasets/
āāā ...
...
Here we have also collected the download links of required datasets for you:
- ShapeNet55/34 (for pre-training): [link].
- ScanObjectNN: [link].
- ModelNet40: [link 1] (pre-processed) or [link 2] (raw).
- ModelNet Few-shot: [link].
- ShapeNetPart: [link].
3. Pre-train a point cloud model (e.g. Point-MAE)
To pre-train Point-MAE on ShapeNet training set, you can run the following command:
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/pretrain.yaml --exp_name <output_file_name>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/pretrain.yaml --exp_name pretrain_pointmae
If you want to try other models or change pre-training configuration, e.g., mask ratios, just create a new configuration file and pass its path to --config.
For a quick start, we also have provided the pre-trained checkpoint of Point-MAE [link].
4. Tune pre-trained point cloud models on downstream tasks
We take VPT and IDPT as two showcases of prompt tuning for pre-trained point cloud models. Executable commands of different downstream tasks are listed below.
4.1 Object Classification
4.1.1 ModelNet40
<details><summary>VPT-Deep (click to expand š)</summary># CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_modelnet_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name modelnet_vpt
# further enable voting mechanism
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet.yaml --test --vote --exp_name modelnet_vpt_vote --ckpts ./experiments/finetune_modelnet/cfgs/modelnet_vpt/ckpt-best.pth
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_modelnet_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name modelnet_idpt
# further enable voting mechanism
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet.yaml --test --vote --exp_name modelnet_idpt_vote --ckpts ./experiments/finetune_modelnet/cfgs/modelnet_idpt/ckpt-best.pth
4.1.2 ScanObjectNN (OBJ-BG)
<details><summary>VPT-Deep (click to expand š)</summary># CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objbg_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objbg_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name bg_vpt
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objbg_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objbg_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name bg_idpt
4.1.3 ScanObjectNN (OBJ-ONLY)
<details><summary>VPT-Deep (click to expand š)</summary># CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objonly_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objonly_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name only_vpt
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_objonly_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objonly_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name only_idpt
4.1.4 ScanObjectNN (PB-T50-RS)
<details><summary>VPT-Deep (click to expand š)</summary># CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_hardest_vpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_hardest_vpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name hard_vpt
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/finetune_scan_hardest_idpt.yaml --finetune_model --exp_name <output_file_name> --ckpts <path/to/pre-trained/model>
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_hardest_idpt.yaml --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --finetune_model --exp_name hard_idpt
4.2 Few-shot Learning on ModelNet Few-shot
<details><summary>VPT-Deep (click to expand š)</summary># CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/fewshot_vpt.yaml --finetune_model --ckpts <path/to/pre-trained/model> --exp_name <output_file_name> --way <5 or 10> --shot <10 or 20> --fold <0-9>
for WAY in 5 10
do
for SHOT in 10 20
do
for FOLD in $(seq 0 9)
do
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/fewshot_vpt.yaml --finetune_model --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --exp_name fewshot_vpt --way ${WAY} --shot ${SHOT} --fold ${FOLD}
done
done
done
# CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/fewshot_idpt.yaml --finetune_model --ckpts <path/to/pre-trained/model> --exp_name <output_file_name> --way <5 or 10> --shot <10 or 20> --fold <0-9>
for WAY in 5 10
do
for SHOT in 10 20
do
for FOLD in $(seq 0 9)
do
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/fewshot_idpt.yaml --finetune_model --ckpts ./checkpoint/pretrain/mae/ckpt-last.pth --exp_name fewshot_idpt --way ${WAY} --shot ${SHOT} --fold ${FOLD}
done
done
done
4.3 Part Segmentation on ShapeNet-Part
<details><summary>IDPT (click to expand š)</summary>cd segmentation
# python main.py --model prompt_pt3 --optimizer_part only_new --ckpts <path/to/pre-trained/model> --root path/to/data --learning_rate 0.0002 --epoch 300
CUDA_VISIBLE_DEVICES=0 python main.py --model prompt_pt3 --optimizer_part only_new --ckpts ../checkpoint/pretrain/mae/ckpt-last.pth --root ../data/shapenetcore_partanno_segmentation_benchmark_v0_normal/ --log_dir seg_idpt --learning_rate 0.0002 --epoch 300
5. Validate with checkpoints
For reproducibility, logs and checkpoints of tuned models via IDPT can be found in the table below.
| Task | Dataset | Trainable Parameters | log | Acc. | Checkpoints Download |
|---|---|---|---|---|---|
| Pre-training | ShapeNet | 1.7M | - | N.A. | Point-MAE |
| Classification | ScanObjectNN | 1.7M | finetune_scan_objbg.log | 93.63% | OBJ-BG |
| Classification | ScanObjectNN | 1.7M | finetune_scan_objonly.log | 93.12% | OBJ-ONLY |
| Classification | ScanObjectNN | 1.7M | finetune_scan_hardest.log | 88.51% | PB-T50-RS |
| Classification | ModelNet40 | 1.7M | finetune_modelnet.log | 93.3% | ModelNet-1k |
| Classification | ModelNet40 (vote) | - | finetune_modelnet_vote.log | 94.4% | - |
| Task | Dataset | log | 5w10s (%) | 5w20s (%) | 10w10s (%) | 10w20s (%) |
|---|---|---|---|---|---|---|
| Few-shot learning | ModelNet40 | fewshot_logs | 97.3 Ā± 2.1 | 97.9 Ā± 1.1 | 92.8 Ā± 4.1 | 95.4 Ā± 2.9 |
The evaluation commands with checkpoints should be in the following format:
CUDA_VISIBLE_DEVICES=<GPU> python main.py --test --config <yaml_file_name> --exp_name <output_file_name> --ckpts <path/to/ckpt>
# object classification on ScanObjectNN (PB-T50-RS)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_hardest_idpt.yaml --ckpts ./checkpoint/hardest/ckpt-best.pth --test --exp_name hard_test
# object classification on ScanObjectNN (OBJ-BG)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objbg_idpt.yaml --ckpts ./checkpoint/bg/ckpt-best.pth --test --exp_name bg_test
# object classification on ScanObjectNN (OBJ-ONLY)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_scan_objonly_idpt.yaml --ckpts ./checkpoint/only/ckpt-best.pth --test --exp_name only_test
# object classification on ModelNet40 (w/o voting)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_idpt.yaml --ckpts ./checkpoint/modelnet40/ckpt-best.pth --test --exp_name model_test
# object classification on ModelNet40 (w/ voting)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/finetune_modelnet_idpt.yaml --test --vote --exp_name modelnet_idpt_vote --ckpts ./checkpoint/modelnet40/ckpt-best.pth
# few-show learning on ModelNet40
python parse_test_res.py ./experiments/all/fewshot --multi-exp --few-shot
6. t-SNE visualization
We used t-SNE to visualize the results obtained by IDPT on the ModelNet40 and ScanObjectNN test sets. The execution commands are as follows:
CUDA_VISIBLE_DEVICES=<GPU> python main.py --tsne --config <yaml_file_name> --exp_name <output_file_name> --ckpts <path/to/ckpt>
# t-SNE on ScanObjectNN (PB-T50-RS)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_scan_hardest_idpt_tsne.yaml --ckpts ./checkpoint/hardest/ckpt-best.pth --tsne --exp_name hard_test_tsne
# object classification on ScanObjectNN (OBJ-BG)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_scan_objbg_idpt_tsne.yaml --ckpts ./checkpoint/bg/ckpt-best.pth --tsne --exp_name bg_test_tsne
# object classification on ScanObjectNN (OBJ-ONLY)
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_scan_objonly_idpt_tsne.yaml --ckpts ./checkpoint/only/ckpt-best.pth --tsne --exp_name only_test_tsne
# object classification on ModelNet40
CUDA_VISIBLE_DEVICES=0 python main.py --config cfgs/tsne/finetune_modelnet_idpt_tsne.yaml --ckpts ./checkpoint/modelnet40/ckpt-best.pth --tsne --exp_name model_test_tsne
You can also modify the code yourself to implement a customized t-SNE visualization. š¤
7. Bibliography
If you find this code useful or use the toolkit in your work, please consider citing:
@inproceedings{zha2023_IDPT,
title={Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models},
author={Zha, Yaohua and Wang, Jinpeng and Dai, Tao and Chen, Bin and Wang, Zhi and Xia, Shu-Tao},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
year={2023}
}
8. Acknowledgements
Our codes are built upon Point-MAE, Point-BERT, ACT, DGCNN and VPT. Thanks for their efforts.
9. Contact
If you have any question, you can raise an issue or email Yaohua Zha (chayh21@mails.tsinghua.edu.cn) and Jinpeng Wang (wjp20@mails.tsinghua.edu.cn). We will reply you soon.