Awesome

ARTrack

The official PyTorch implementation of our CVPR 2023 Highlight and CVPR 2024 Poster paper:

Autoregressive Visual Tracking

ARTrackV2: Prompting Autoregressive Tracker Where to Look and How to Describe

GitHub maintainer: Yifan Bai

[CVPR2023] [CVPR2024]

:bookmark: Update List:

• Fastitnp
• VastTrack and More Datasets

:bookmark: Checkpoints in Baidu Pan

[Baidu Pan] passwd: g06j

Google Drive: ARTrackV2-B-256 | ARTrackV2-B-256-GOT | ARTrackV2-L-384-GOT |

:bookmark:Our ARTrackV2 is accepted by CVPR2024!!!

[Deprecated Code] [Raw Result]

We have released the training code for ARTrackV2 and merged it into this repository. You can adjust the config to use the corresponding ARTrackV2-marked modules. The training process remains the same as V1.

Highlight

:bookmark:Brief Introduction

We present ARTrack, an autoregressive framework for visual object tracking. ARTrack tackles tracking as a coordinate sequence interpretation task that estimates object trajectories progressively, where the current estimate is induced by previous states and in turn affects subsequences. This time-autoregressive approach models the sequential evolution of trajectories to keep tracing the object across frames, making it superior to existing template matching based trackers that only consider the per-frame localization accuracy. ARTrack is simple and direct, eliminating customized localization heads and post-processings. Despite its simplicity, ARTrack achieves state-of-the-art performance on prevailing benchmark datasets.

:bookmark:Strong Performance

:bookmark:Inference Speed

Our baseline model (backbone: ViT-B, resolution: 256x256) can run at 26 fps (frames per second) on a single NVIDIA GeForce RTX 3090, our alter decoder version can run at 45 fps on a single NVIDIA GeForce RTX 3090.

Bug of array of inhomogeneous shape

Thanks to MrtXue, if you meet the "ValueError: setting an array element with a sequence." when you train in the second stage, you can try to reduce your numpy version to 1.23.

Update for checkpoint(ARTrack_large_384_full):

You can download the model weights from Google Drive

Update for checkpoint and raw_result(ARTrack_base_256_full):

You can download the model weights and raw_result from Google Drive

Install the environment

Use the Anaconda (CUDA 11.3)

conda env create -f ARTrack_env_cuda113.yaml

Set project paths

Run the following command to set paths for this project

python tracking/create_default_local_file.py --workspace_dir . --data_dir ./data --save_dir ./output

After running this command, you can also modify paths by editing these two files

lib/train/admin/local.py  # paths about training
lib/test/evaluation/local.py  # paths about testing

Data Preparation

Put the tracking datasets in ./data. It should look like this:

${PROJECT_ROOT}
 -- data
     -- lasot
         |-- airplane
         |-- basketball
         |-- bear
         ...
     -- got10k
         |-- test
         |-- train
         |-- val
     -- coco
         |-- annotations
         |-- images
     -- trackingnet
         |-- TRAIN_0
         |-- TRAIN_1
         ...
         |-- TRAIN_11
         |-- TEST

Training

Download pre-trained MAE ViT-Base weights and put it under $PROJECT_ROOT$/pretrained_models (different pretrained models can also be used, see MAE for more details).

One-stage pair-level training

Since sequence-level training requires video input, and the COCO dataset contains only images, traditional training methods were first used to train the model so that it could be fairly compared to other trackers.

python tracking/train.py --script artrack --config artrack_256_full --save_dir ./output --mode multiple --nproc_per_node 4 --use_wandb 0

Replace --config with the desired model config under experiments/artrack. We use wandb to record detailed training logs, in case you don't want to use wandb, set --use_wandb 0.

Two-stage sequence-level training

To enable sequence-level training, replace 'experience/artrack_seq/*.yaml' PRETRAIN_PTH in the yaml configuration file with the path to your pretrained checkpoint, such as './output/artrack_256_full/checkpoints/train/artrack/artrack_256_full/ARTrack_ep0240.pth.tar'.

python tracking/train.py --script artrack_seq --config artrack_seq_256_full --save_dir ./output --mode multiple --nproc_per_node 4 --use_wandb 0

Evaluation

Change the corresponding values of lib/test/evaluation/local.py to the actual benchmark saving paths

Some testing examples:

• LaSOT or other off-line evaluated benchmarks (modify --dataset correspondingly)

python tracking/test.py artrack_seq artrack_seq_256_full --dataset lasot --threads 16 --num_gpus 4
python tracking/analysis_results.py # need to modify tracker configs and names

• GOT10K-test

python tracking/test.py artrack_seq artrack_seq_256_full --dataset got10k_test --threads 16 --num_gpus 4
python lib/test/utils/transform_got10k.py --tracker_name ostrack --cfg_name vitb_384_mae_ce_32x4_got10k_ep100

• TrackingNet

python tracking/test.py artrack_seq artrack_seq_256_full --dataset trackingnet --threads 16 --num_gpus 4
python lib/test/utils/transform_trackingnet.py --tracker_name ostrack --cfg_name vitb_384_mae_ce_32x4_ep300

Acknowledgement

:heart::heart::heart:Our idea is implemented base on the following projects. We really appreciate their excellent open-source works!

• SIoU [related paper]
• OSTrack [related paper]
• PyTracking [related paper]

:heart::heart::heart:This project is not for commercial use. For commercial use, please contact the author.

:heart::heart::heart:This project is not for commercial use. For commercial use, please contact the author.

:heart::heart::heart:This project is not for commercial use. For commercial use, please contact the author.

Citation

If any parts of our paper and code help your research, please consider citing us and giving a star to our repository.

@InProceedings{Wei_2023_CVPR,
    author    = {Wei, Xing and Bai, Yifan and Zheng, Yongchao and Shi, Dahu and Gong, Yihong},
    title     = {Autoregressive Visual Tracking},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {9697-9706}
}
@InProceedings{Bai_2024_CVPR,
    author    = {Bai, Yifan and Zhao, Zeyang and Gong, Yihong and Wei, Xing},
    title     = {ARTrackV2: Prompting Autoregressive Tracker Where to Look and How to Describe},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024}
}

Contact

If you have any questions or concerns, feel free to open issues or directly contact me through the ways on my GitHub homepage provide below paper's title.