Awesome

TransTrack: Multiple Object Tracking with Transformer

Introduction

TransTrack: Multiple Object Tracking with Transformer

Updates

• (22/02/2022) Multi-GPU testing is supported.
• (29/10/2021) Automatic Mixed Precision(AMP) training is supported.
• (28/04/2021) Higher performance is reported by training on mixture of CrowdHuman and MOT, instead of first CrowdHuman then MOT.
• (28/04/2021) Higher performance is reported by pre-training both detection and tracking on CrowdHuman, instead of only detection.
• (28/04/2021) Higher performance is reported by increasing the number of queries from 300 to 500.
• (08/04/2021) Refactoring the code.

MOT challenge

Validation set

If download link is invalid, models and logs are also available in Github Release and Baidu Drive by code m4iv.

Notes

• We observe about 1 MOTA noise.
• If the resulting MOTA of your self-trained model is not desired, playing around with the --track_thresh sometimes gives a better performance.
• The default track_thresh is 0.4, except for 0.5 in crowdhuman.
• The training time is on 8 NVIDIA V100 GPUs with batchsize 16.
• We use the models pre-trained on imagenet.
• (crowdhuman, mot17_half) is first training on crowdhuman, then fine-tuning on mot17_half.

Demo

<img src="assets/MOT17-11.gif" width="400"/> <img src="assets/MOT17-04.gif" width="400"/>

Installation

The codebases are built on top of Deformable DETR and CenterTrack.

Requirements

• Linux, CUDA>=9.2, GCC>=5.4
• Python>=3.7
• PyTorch ≥ 1.5 and torchvision that matches the PyTorch installation. You can install them together at pytorch.org to make sure of this
• OpenCV is optional and needed by demo and visualization

Steps

1. Install and build libs

git clone https://github.com/PeizeSun/TransTrack.git
cd TransTrack
cd models/ops
python setup.py build install
cd ../..
pip install -r requirements.txt

2. Prepare datasets and annotations

mkdir crowdhuman
cp -r /path_to_crowdhuman_dataset/CrowdHuman_train crowdhuman/CrowdHuman_train
cp -r /path_to_crowdhuman_dataset/CrowdHuman_val crowdhuman/CrowdHuman_val
mkdir mot
cp -r /path_to_mot_dataset/train mot/train
cp -r /path_to_mot_dataset/test mot/test

CrowdHuman dataset is available in CrowdHuman.

python3 track_tools/convert_crowdhuman_to_coco.py

MOT dataset is available in MOT.

python3 track_tools/convert_mot_to_coco.py

3. Pre-train on crowdhuman

sh track_exps/crowdhuman_train.sh
python3 track_tools/crowdhuman_model_to_mot.py

The pre-trained model is available crowdhuman_final.pth.

4. Train TransTrack

sh track_exps/crowdhuman_mot_trainhalf.sh

5. Evaluate TransTrack

sh track_exps/mot_val.sh
sh track_exps/mota.sh

6. Visualize TransTrack

python3 track_tools/txt2video.py

Test set

Notes

• Performance on test set is evaluated by MOT challenge.
• (crowdhuman + mot17) is training on mixture of crowdhuman and mot17.
• We won't release trained models for test test. Running as in Steps could reproduce them.

Steps

1. Train TransTrack

sh track_exps/crowdhuman_mot_train.sh

or

1. Mix crowdhuman and mot17

mkdir -p mix/annotations
cp mot/annotations/val_half.json mix/annotations/val_half.json
cp mot/annotations/test.json mix/annotations/test.json
cd mix
ln -s ../mot/train mot_train
ln -s ../crowdhuman/CrowdHuman_train crowdhuman_train
cd ..
python3 track_tools/mix_data.py

2. Train TransTrack

sh track_exps/crowdhuman_plus_mot_train.sh

License

TransTrack is released under MIT License.

Citing

If you use TransTrack in your research or wish to refer to the baseline results published here, please use the following BibTeX entries:

@article{transtrack,
  title   =  {TransTrack: Multiple-Object Tracking with Transformer},
  author  =  {Peize Sun and Jinkun Cao and Yi Jiang and Rufeng Zhang and Enze Xie and Zehuan Yuan and Changhu Wang and Ping Luo},
  journal =  {arXiv preprint arXiv: 2012.15460},
  year    =  {2020}
}