Awesome
SiamMask
NEW: now including code for both training and inference!
This is the official implementation with training code for SiamMask (CVPR2019). For technical details, please refer to:
Fast Online Object Tracking and Segmentation: A Unifying Approach <br /> Qiang Wang*, Li Zhang*, Luca Bertinetto*, Weiming Hu, Philip H.S. Torr (* denotes equal contribution) <br /> CVPR 2019 <br /> [Paper] [Video] [Project Page] <br />
<div align="center"> <img src="http://www.robots.ox.ac.uk/~qwang/SiamMask/img/SiamMask.jpg" width="600px" /> </div>Bibtex
If you find this code useful, please consider citing:
@inproceedings{wang2019fast,
title={Fast online object tracking and segmentation: A unifying approach},
author={Wang, Qiang and Zhang, Li and Bertinetto, Luca and Hu, Weiming and Torr, Philip HS},
booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
year={2019}
}
Contents
Environment setup
This code has been tested on Ubuntu 16.04, Python 3.6, Pytorch 0.4.1, CUDA 9.2, RTX 2080 GPUs
- Clone the repository
git clone https://github.com/foolwood/SiamMask.git && cd SiamMask
export SiamMask=$PWD
- Setup python environment
conda create -n siammask python=3.6
source activate siammask
pip install -r requirements.txt
bash make.sh
- Add the project to your PYTHONPATH
export PYTHONPATH=$PWD:$PYTHONPATH
Demo
- Setup your environment
- Download the SiamMask model
cd $SiamMask/experiments/siammask_sharp
wget http://www.robots.ox.ac.uk/~qwang/SiamMask_VOT.pth
wget http://www.robots.ox.ac.uk/~qwang/SiamMask_DAVIS.pth
- Run
demo.py
cd $SiamMask/experiments/siammask_sharp
export PYTHONPATH=$PWD:$PYTHONPATH
python ../../tools/demo.py --resume SiamMask_DAVIS.pth --config config_davis.json
<div align="center">
<img src="http://www.robots.ox.ac.uk/~qwang/SiamMask/img/SiamMask_demo.gif" width="500px" />
</div>
Testing
- Setup your environment
- Download test data
cd $SiamMask/data
sudo apt-get install jq
bash get_test_data.sh
- Download pretrained models
cd $SiamMask/experiments/siammask_sharp
wget http://www.robots.ox.ac.uk/~qwang/SiamMask_VOT.pth
wget http://www.robots.ox.ac.uk/~qwang/SiamMask_VOT_LD.pth
wget http://www.robots.ox.ac.uk/~qwang/SiamMask_DAVIS.pth
- Evaluate performance on VOT
bash test_mask_refine.sh config_vot.json SiamMask_VOT.pth VOT2016 0
bash test_mask_refine.sh config_vot.json SiamMask_VOT.pth VOT2018 0
bash test_mask_refine.sh config_vot.json SiamMask_VOT.pth VOT2019 0
bash test_mask_refine.sh config_vot18.json SiamMask_VOT_LD.pth VOT2016 0
bash test_mask_refine.sh config_vot18.json SiamMask_VOT_LD.pth VOT2018 0
python ../../tools/eval.py --dataset VOT2016 --tracker_prefix C --result_dir ./test/VOT2016
python ../../tools/eval.py --dataset VOT2018 --tracker_prefix C --result_dir ./test/VOT2018
python ../../tools/eval.py --dataset VOT2019 --tracker_prefix C --result_dir ./test/VOT2019
- Evaluate performance on DAVIS (less than 50s)
bash test_mask_refine.sh config_davis.json SiamMask_DAVIS.pth DAVIS2016 0
bash test_mask_refine.sh config_davis.json SiamMask_DAVIS.pth DAVIS2017 0
- Evaluate performance on Youtube-VOS (need download data from website)
bash test_mask_refine.sh config_davis.json SiamMask_DAVIS.pth ytb_vos 0
Results
These are the reproduction results from this repository. All results can be downloaded from our project page.
<sub>Tracker</sub> | <sub>VOT2016</br>EAO / A / R</sub> | <sub>VOT2018</br>EAO / A / R</sub> | <sub>DAVIS2016</br>J / F</sub> | <sub>DAVIS2017</br>J / F</sub> | <sub>Youtube-VOS</br>J_s / J_u / F_s / F_u</sub> | <sub>Speed</sub> |
---|---|---|---|---|---|---|
<sub>SiamMask-box</sub> | <sub>0.412/0.623/0.233</sub> | <sub>0.363/0.584/0.300</sub> | - / - | - / - | - / - / - / - | <sub>77 FPS</sub> |
<sub>SiamMask</sub> | <sub>0.433/0.639/0.214</sub> | <sub>0.380/0.609/0.276</sub> | <sub>0.713/0.674</sub> | <sub>0.543/0.585</sub> | <sub>0.602/0.451/0.582/0.477</sub> | <sub>56 FPS</sub> |
<sub>SiamMask-LD</sub> | <sub>0.455/0.634/0.219</sub> | <sub>0.423/0.615/0.248</sub> | - / - | - / - | - / - / - / - | <sub>56 FPS</sub> |
Note:
- Speed are tested on a NVIDIA RTX 2080.
-box
reports an axis-aligned bounding box from the box branch.-LD
means training with large dataset (ytb-bb+ytb-vos+vid+coco+det).
Training
Training Data
- Download the Youtube-VOS, COCO, ImageNet-DET, and ImageNet-VID.
- Preprocess each datasets according the readme files.
Download the pre-trained model (174 MB)
(This model was trained on the ImageNet-1k Dataset)
cd $SiamMask/experiments
wget http://www.robots.ox.ac.uk/~qwang/resnet.model
ls | grep siam | xargs -I {} cp resnet.model {}
Training SiamMask base model
- Setup your environment
- From the experiment directory, run
cd $SiamMask/experiments/siammask_base/
bash run.sh
- Training takes about 10 hours in our 4 Tesla V100 GPUs.
- If you experience out-of-memory errors, you can reduce the batch size in
run.sh
. - You can view progress on Tensorboard (logs are at <experiment_dir>/logs/)
- After training, you can test checkpoints on VOT dataset.
bash test_all.sh -s 1 -e 20 -d VOT2018 -g 4 # test all snapshots with 4 GPUs
- Select best model for hyperparametric search.
#bash test_all.sh -m [best_test_model] -d VOT2018 -n [thread_num] -g [gpu_num] # 8 threads with 4 GPUS
bash test_all.sh -m snapshot/checkpoint_e12.pth -d VOT2018 -n 8 -g 4 # 8 threads with 4 GPUS
Training SiamMask model with the Refine module
- Setup your environment
- In the experiment file, train with the best SiamMask base model
cd $SiamMask/experiments/siammask_sharp
bash run.sh <best_base_model>
bash run.sh checkpoint_e12.pth
- You can view progress on Tensorboard (logs are at <experiment_dir>/logs/)
- After training, you can test checkpoints on VOT dataset
bash test_all.sh -s 1 -e 20 -d VOT2018 -g 4
Training SiamRPN++ model (unofficial)
- Setup your environment
- From the experiment directory, run
cd $SiamMask/experiments/siamrpn_resnet
bash run.sh
- You can view progress on Tensorboard (logs are at <experiment_dir>/logs/)
- After training, you can test checkpoints on VOT dataset
bash test_all.sh -h
bash test_all.sh -s 1 -e 20 -d VOT2018 -g 4
License
Licensed under an MIT license.