Awesome
SeqTrackv2
Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking<br> Xin Chen, Ben Kang, Jiawen Zhu, Dong Wang, Houwen Peng, Huchuan Lu
This is an official pytorch implementation of the paper Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking.
Highlights
Seq2seq modeling
SeqTrack models tracking as a sequence generation task. If the model knows where the target object is, we could simply teach it how to read the bounding box out.
Simple architecture and loss function
SeqTrack only adopts a plain encoder-decoder transformer architecture with a simple cross-entropy loss.
Unified framework for multi-modal tracking
SeqTrackv2 handles four multi-modal tracking tasks with a unified model and parameter set.
Strong performance
Single-Modal Tracking:
Tracker | LaSOT (AUC) | GOT-10K (AO) | TrackingNet (AUC) |
---|---|---|---|
SeqTrack | 72.5 | 74.8 | 85.5 |
OSTrack | 71.1 | 73.7 | 83.9 |
SimTrack | 70.5 | 69.8 | 83.4 |
Mixformer | 70.1 | 70.7 | 83.9 |
Multi-Modal Tracking:
Tracker | LasHeR (AUC) | RGBT234 (MSR) | VOT-RGBD22 (EAO) | DepthTrack (F-score) | VisEvent (AUC) | TNL2K (AUC) |
---|---|---|---|---|---|---|
SeqTrackv2 | 61.0 | 68.0 | 74.8 | 62.3 | 63.4 | 62.4 |
ViPT | 52.5 | 61.7 | 72.1 | 59.4 | 59.2 | - |
Un-Track | - | 62.5 | 72.1 | 61.0 | 58.9 | - |
JointNLT | - | - | - | - | - | 56.9 |
Install the environment
conda create -n seqtrackv2 python=3.8
conda activate seqtrackv2
bash install.sh
- Add the project path to environment variables
export PYTHONPATH=<absolute_path_of_SeqTrackv2>:$PYTHONPATH
Data Preparation
Put the tracking datasets in ./data. It should look like:
For RGB tracking:
${SeqTrack_ROOT}
-- data
-- lasot
|-- airplane
|-- basketball
|-- bear
...
-- got10k
|-- test
|-- train
|-- val
-- coco
|-- annotations
|-- images
-- trackingnet
|-- TRAIN_0
|-- TRAIN_1
...
|-- TRAIN_11
|-- TEST
For Multi-Modal tracking (RGB-T234 and otb_lang is only for evaluation, thus can be ignored):
${SeqTrack_ROOT}
-- data
-- depthtrack
-- train
|-- adapter02_indoor
|-- bag03_indoor
|-- bag04_indoor
...
-- lasher
-- trainingset
|-- 1boygo
|-- 1handsth
|-- 1phoneblue
...
-- testingset
|-- 1blackteacher
|-- 1boycoming
|-- 1stcol4thboy
...
-- RGB-T234
|-- afterrain
|-- aftertree
|-- baby
...
-- visevent
-- train
|-- 00142_tank_outdoor2
|-- 00143_tank_outdoor2
|-- 00144_tank_outdoor2
...
-- test
|-- 00141_tank_outdoor2
|-- 00147_tank_outdoor2
|-- 00197_driving_outdoor3
...
-- annos
-- tnl2k
-- train
|-- Arrow_Video_ZZ04_done
|-- Assassin_video_1-Done
|-- Assassin_video_2-Done
...
-- test
|-- advSamp_Baseball_game_002-Done
|-- advSamp_Baseball_video_01-Done
|-- advSamp_Baseball_video_02-Done
...
-- lasot
|-- airplane
|-- basketball
|-- bear
...
-- refcoco
-- images
-- train2014
|-- COCO_train2014_000000291797.jpg
|-- COCO_train2014_000000581921.jpg
|-- COCO_train2014_000000291788.jpg
...
-- refcocog
|-- instances.json
-- otb_lang
-- OTB_videos
|-- Basketball
|-- Biker
|-- Bird1
...
-- OTB_query_train
|-- Basketball.txt
|-- Bolt.txt
|-- Boy.txt
...
-- OTB_query_test
|-- Biker.txt
|-- Bird1.txt
|-- Bird2.txt
...
Set project paths
Run the following command to set paths for this project
python tracking/create_default_local_file.py --workspace_dir . --data_dir ./data --save_dir .
After running this command, you can also modify paths by editing these two files
lib/train/admin/local.py # paths about training
lib/test/evaluation/local.py # paths about testing
Train
Train SeqTrack
python -m torch.distributed.launch --nproc_per_node 8 lib/train/run_training.py --script seqtrack --config seqtrack_b256 --save_dir .
(Optionally) Debugging training with a single GPU
python tracking/train.py --script seqtrack --config seqtrack_b256 --save_dir . --mode single
Train SeqTrackv2
The pretrained SeqTrack models and Bert models can be downloaded here [Google Drive][Baidu Drive]. Put the pretrained models in ./pretrained, it should be like:
${SeqTrack_ROOT}
-- pretrained
-- bert
|-- bert-base-uncased.tar.gz
|-- bert-base-uncased-vocab.txt
|-- bert_config.json
|-- pytorch_model.bin
-- seqtrack
-- seqtrack_b256
|-- SEQTRACK_ep0500.pth.tar
-- seqtrack_b384
|-- SEQTRACK_ep0500.pth.tar
-- seqtrack_l256
|-- SEQTRACK_ep0500.pth.tar
-- seqtrack_l384
|-- SEQTRACK_ep0500.pth.tar
Then, run the following command:
python -m torch.distributed.launch --nproc_per_node 4 lib/train/run_training.py --script seqtrackv2 --config seqtrackv2_b256 --save_dir .
(Optionally) Debugging training with a single GPU
python tracking/train.py --script seqtrackv2 --config seqtrackv2_b256 --save_dir . --mode single
Test and evaluate on benchmarks
SeqTrack for RGB-based Tracking
- LaSOT
python tracking/test.py seqtrack seqtrack_b256 --dataset lasot --threads 2
python tracking/analysis_results.py # need to modify tracker configs and names
- GOT10K-test
python tracking/test.py seqtrack seqtrack_b256_got --dataset got10k_test --threads 2
python lib/test/utils/transform_got10k.py --tracker_name seqtrack --cfg_name seqtrack_b256_got
- TrackingNet
python tracking/test.py seqtrack seqtrack_b256 --dataset trackingnet --threads 2
python lib/test/utils/transform_trackingnet.py --tracker_name seqtrack --cfg_name seqtrack_b256
- TNL2K
python tracking/test.py seqtrack seqtrack_b256 --dataset tnl2k --threads 2
python tracking/analysis_results.py # need to modify tracker configs and names
- UAV123
python tracking/test.py seqtrack seqtrack_b256 --dataset uav --threads 2
python tracking/analysis_results.py # need to modify tracker configs and names
- NFS
python tracking/test.py seqtrack seqtrack_b256 --dataset nfs --threads 2
python tracking/analysis_results.py # need to modify tracker configs and names
- VOT2020
Before evaluating "SeqTrack+AR" on VOT2020, please install some extra packages following external/AR/README.md
cd external/vot20/<workspace_dir>
export PYTHONPATH=<path to the seqtrack project>:$PYTHONPATH
vot evaluate --workspace . seqtrack_b256_ar
vot analysis --nocache
SeqTrackv2 for Multi-Modal Tracking
- LasHeR
python ./RGBT_workspace/test_rgbt_mgpus.py --script_name seqtrackv2 --dataset_name LasHeR --yaml_name seqtrackv2_b256
# Through this command, you can obtain the tracking result. Then, please use the official matlab toolkit to evaluate the result.
- RGBT-234
python ./RGBT_workspace/test_rgbt_mgpus.py --script_name seqtrackv2 --dataset_name RGBT234 --yaml_name seqtrackv2_b256
# Through this command, you can obtain the tracking result. Then, please use the official matlab toolkit to evaluate the result.
- VisEvent
python ./RGBE_workspace/test_rgbe_mgpus.py --script_name seqtrackv2 --dataset_name VisEvent --yaml_name seqtrackv2_b256
# Through this command, you can obtain the tracking result. Then, please use the official matlab toolkit to evaluate the result.
- TNL2K
python tracking/test.py seqtrackv2 seqtrackv2_b256 --dataset tnl2k --threads 0
python tracking/analysis_results.py # need to modify tracker configs and names
- OTB99
python tracking/test.py seqtrackv2 seqtrackv2_b256 --dataset otb99_lang --threads 0
python tracking/analysis_results.py # need to modify tracker configs and names
- DepthTrack
cd Depthtrack_workspace
vot evaluate seqtrackv2_b256
vot analysis seqtrackv2_b256 --nocache
- VOT-RGBD22
cd VOT22RGBD_workspace
vot evaluate seqtrackv2_b256
vot analysis seqtrackv2_b256 --nocache
Model Zoo
The trained models, and the raw tracking results are provided in the model zoo
Contact
- Xin Chen (email:chenxin3131@mail.dlut.edu.cn)