Home

Awesome

<div align="center"> <h1>Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learning🚀</h1>

Dingkang Liang<sup>1</sup>* ,Tianrui Feng<sup>1</sup>* ,Xin Zhou<sup>1</sup>* , Yumeng Zhang<sup>2</sup>, Zhikang Zou<sup>2</sup>, and Xiang Bai<sup> 1✉️</sup>

<sup>1</sup> Huazhong University of Science and Technology, <sup>2</sup> Baidu Inc.

(*) equal contribution, (​✉️​) corresponding author.

arXiv Code License PWC PWC PWC PWC

</div>

News

[2024-10-10] PointGST is released.

Abstract

Recently, leveraging pre-training techniques to enhance point cloud models has become a hot research topic. However, existing approaches typically require full fine-tuning of pre-trained models to achieve satisfied performance on downstream tasks, accompanying storage-intensive and computationally demanding. To address this issue, we propose a novel Parameter-Efficient Fine-Tuning (PEFT) method for point cloud, called PointGST (Point cloud Graph Spectral Tuning). PointGST freezes the pre-trained model and introduces a lightweight, trainable Point Cloud Spectral Adapter (PCSA) to fine-tune parameters in the spectral domain.

<div align="center"> <img src="./figure/intro.png" width = "888" align=center /> </div>

Extensive experiments on challenging point cloud datasets across various tasks demonstrate that PointGST not only outperforms its fully fine-tuning counterpart but also significantly reduces trainable parameters, making it a promising solution for efficient point cloud learning. More importantly, it improves upon a solid baseline by +2.28%, 1.16%, and 2.78%, resulting in 99.48%, 97.76%, and 96.18% on the ScanObjNN OBJ_BG, OBJ_OBLY, and PB_T50_RS datasets, respectively. This advancement establishes a new state-of-the-art, using only 0.67% of the trainable parameters.

Overview

<div align="center"> <img src="./figure/pipeline.png" width = "888" align=center /> </div>

Getting Started

Installation

We recommend using Anaconda for the installation process:

git clone https://github.com/jerryfeng2003/PointGST.git
cd PointGST/

Requirements

conda create -y -n pgst python=3.9
conda activate pgst
pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

# Chamfer Distance & emd
cd ./extensions/chamfer_dist
python setup.py install --user
cd ../emd
python setup.py install --user

# PointNet++
pip install "git+https://github.com/erikwijmans/Pointnet2_PyTorch.git#egg=pointnet2_ops&subdirectory=pointnet2_ops_lib"

# GPU kNN
pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl

Datasets

See DATASET.md for details.

Main Results

<div align="center"> <img src="./figure/result.png" width = "888" align=center /> </div> <div align="center"> <img src="./figure/result2.png" width = "888" align=center /> </div>
BaselineTrainable ParametersDatasetConfigAcc.Download
Point-MAE <br> (ECCV 22)0.6MModelNet40 <br> OBJ_BG <br> OBJ_ONLY <br> PB_T50_RSmodelnet <br> scan_objbg <br> scan_objonly <br> scan_hardest93.5 <br> 91.74 <br> 90.19 <br> 85.29ckpt <br> ckpt <br> ckpt <br> ckpt
ACT <br> (ICLR 23)0.6MModelNet40 <br> OBJ_BG <br> OBJ_ONLY <br> PB_T50_RSmodelnet <br> scan_objbg <br> scan_objonly <br> scan_hardest93.4 <br> 93.46 <br> 92.60 <br> 88.27ckpt <br> ckpt <br> ckpt <br> ckpt
ReCon <br> (ICML 23)0.6MModelNet40 <br> OBJ_BG <br> OBJ_ONLY <br> PB_T50_RSmodelnet <br> scan_objbg <br> scan_objonly <br> scan_hardest93.6 <br> 94.49 <br> 92.94 <br> 89.49ckpt <br> ckpt <br> ckpt <br> ckpt
PointGPT-L <br> (NeurIPS 24)2.4MModelNet40 <br> OBJ_BG <br> OBJ_ONLY <br> PB_T50_RSmodelnet <br> scan_objbg <br> scan_objonly <br> scan_hardest94.8 <br> 98.97 <br> 97.59 <br> 94.83ckpt <br> ckpt <br> ckpt <br> ckpt
PointGPT-L (voting) <br> (NeurIPS 24)2.4MModelNet40 <br> OBJ_BG <br> OBJ_ONLY <br> PB_T50_RSmodelnet <br> scan_objbg <br> scan_objonly <br> scan_hardest95.3 <br> 99.48 <br> 97.76 <br> 96.18log <br> log <br> log <br> log

The evaluation commands with checkpoints should be in the following format:

CUDA_VISIBLE_DEVICES=<GPU> python main.py --test --config <path/to/cfg> --exp_name <path/to/output> --ckpts <path/to/ckpt>

# further enable voting mechanism
CUDA_VISIBLE_DEVICES=<GPU> python main.py --test --vote --config <path/to/cfg> --exp_name <path/to/output> --ckpts <path/to/ckpt>

All the experiments are conducted on a single NVIDIA 3090 GPU.

t-SNE visualization

# t-SNE on ScanObjectNN
CUDA_VISIBLE_DEVICES=<GPU> python main.py --config <path/to/cfg> --ckpts <path/to/ckpt> --tsne --exp_name <path/to/output>

To Do

Acknowledgement

This project is based on Point-BERT (paper, code), Point-MAE (paper, code), ACT(paper, code), ReCon (paper, code), PointGPT(paper, code), IDPT (paper, code), and DAPT(paper, code). Thanks for their wonderful works.

Citation

If you find this repository useful in your research, please consider giving a star ⭐ and a citation.

@article{liang2024pointgst,
  title={Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learning},
  author={Liang, Dingkang and Feng, Tianrui and Zhou, Xin and Zhang, Yumeng and Zou, Zhikang and Bai, Xiang},
  journal={arXiv preprint arXiv:2410.08114},
  year={2024}
}