Home

Awesome

SDSTrack

This is an official release of the CVPR 2024 paper: SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking.

Models & Raw Results(Google Driver) Models & Raw Results(Baidu Driver: qolo)

<p align="center"> <img width="85%" src="assets/pipeline.png" alt="Pipeline"/> </p>

News

[Mar 26, 2024]

✅ We release codes, models and raw results.

[Feb 27, 2024]

✅ SDSTrack is accepted to CVPR2024.

Highlights

💡 SDSTrack uses lightweight adapters for parameter-efficient fine-tuning and can be used for various modality combinations.

💡 We employ a symmetric design, treating all modalities equally, thereby reducing modality dependencies.

💡 We significantly improve the robustness of trackers through complementary masked patch distillation.

💡 SDSTrack performs excellently even in extreme scenarios, such as modality absence and severe occlusion.

Usage

Install the environment

Create and activate a conda environment:

conda create -n sdstrack python=3.8
conda activate sdstrack

Install the required packages:

bash install_sdstrack.sh

Data Preparation

Put the training datasets in ./data/. It should look like:

$<PROJECT_ROOT>
-- data
    -- depthtrack/train
        |-- adapter02_indoor
        |-- bag03_indoor
        ...
    -- lasher/trainingset
        |-- 1boygo
        |-- 1handsth
        ...
    -- visevent/train
        |-- 00142_tank_outdoor2
        |-- 00143_tank_outdoor2
        ...
        |-- trainlist.txt

Set project paths

Run the following command to set paths:

cd <PROJECT_ROOT>
python tracking/create_default_local_file.py --workspace_dir . --data_dir ./data --save_dir ./output

You can also modify paths with these two files if needed:

lib/train/admin/local.py      # paths for training
lib/test/evaluation/local.py  # paths for testing

Training

Download the pretrained foundation model (OSTrack) and put it under $PROJECT_ROOT$/pretrained. To train RGB-Depth tracking, you can run:

bash train_sdstrack_rgbd.sh

You can also train various modality combinations by bash train_sdstrack_rgbe.sh and bash train_sdstrack_rgbt.sh.

Evaluation

Acknowledgments

Citation

If our work is useful for your research, please consider citing:

@InProceedings{Hou_2024_CVPR,
  author    = {Hou, Xiaojun and Xing, Jiazheng and Qian, Yijie and Guo, Yaowei and Xin, Shuo and Chen, Junhao and Tang, Kai and Wang, Mengmeng and Jiang, Zhengkai and Liu, Liang and Liu, Yong},
  title     = {SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2024},
  pages     = {26551-26561}
}