Home

Awesome

AGSS-VOS: Attention Guided Single-Shot Video Object Segmentation (ICCV 2019)

Prerequisites

Model training and evaluation

Data Preparation

  1. Download YouTubeVOS.
  2. Download DAVIS-2017.
  3. We prepare the split annotations for all dataset and meta.json for davis2017 in here
  4. Symlink the corresponding train/validation/test dataset and json files to data folder.
data
├── youtube_vos
│   ├── train
│   │   ├── JPEGImages
│   │   ├── Split_Annotations
│   │   ├── meta.json
│   ├── valid
│   │   ├── JPEGImages
│   │   ├── Split_Annotations
│   │   ├── meta.json
│   ├── valid_all_frames
│   │   ├── JPEGImages
├── davis2017
│   ├── trainval
│   │   ├── JPEGImages
│   │   ├── Split_Annotations
│   │   ├── train_meta.json
│   │   ├── val_meta.json
│   ├── test
│   │   ├── JPEGImages
│   │   ├── Split_Annotations
│   │   ├── test_meta.json

Model Preparation

  1. Download RGMP and place it (weigths.pth) in the 'checkpoints' folder.
  2. Download the pretrained model and place it in the 'checkpoints' folder.
checkpoints
├── weights.pth
├── train_ytv
│   ├── model_4.pth
├── train_davis
│   ├── model_199.pth
├── ft_davis
│   ├── model_99.pth
  1. Download FlowNet2C and place it in the 'flow_inference/models/FlowNet2-C_checkpoint.pth.tar'. You need to run
sh make.sh

in the 'channelnorm_package', 'correlation_package' and 'resample2d_package' in 'flow_inference/networks_flow/' folder. Make sure the version of PyTorch is '0.4.0'.

Training

  1. To train on Youtube-VOS training set.
sh run_ytv.sh
  1. To train on DAVIS-2017 training set.
sh run_davis.sh

The checkpoint will be saved in the 'Outputs' folder.

Finetuning

  1. To finetune on the DAVIS-2017 training set with pretrained model on Youtube-VOS traininset.
sh ft_davis.sh

Validation

  1. To inference on Youtube-VOS validation set.
sh val_ytv.sh
  1. To inference on DAVIS-2017 validation set.
sh val_davis.sh
  1. To inference on DAVIS-2017 test-dev set.
sh test_davis.sh
  1. To inference on DAVIS-2017 validation/test-dev set with finetuned model.
sh val_davis_ft.sh

sh test_davis_ft.sh

The results will be saved in the 'val_dir' or 'test_dir' folder. You can change the '--restore' in scripts to validate your own training result. You can also add '--show-img' in scripts to save the visualized image result.

This software is for Non-commercial Research Purposes only.

Contact

If you have any questions, please feel free to contact the authors.

Huaijia Lin linhj@cse.cuhk.edu.hk or huaijialin@gmail.com

Citation

If you use our code, please consider citing our paper:

@inproceedings{lin2019agss,
  title={AGSS-VOS: Attention Guided Single-Shot Video Object Segmentation},
  author={Lin, Huaijia and Qi, Xiaojuan and Jia, Jiaya},
  booktitle={ICCV},
  year={2019}
}

Acknowledgments

Parts of this code were derived from RGMP and FlowNet2.