Awesome

Pretext-Contrastive Learning: Toward Good Practices in Self-supervised Video Representation Leaning

arXiv paper

Currently support PCL(VCP). For VCOP and 3DRotNet, codes are still in refactoring.

A sentence to conclude this paper: if you are developing novel pretext task-based methods for video self-supervised learning, do not hesitate to combine contrastive learning loss, which is simple to use and can boost the performance.

Highlights

1. This paper represents a joint optimization method in self-supervised video representation learning, which can achieve high performance without proposing new pretext tasks;
2. The effectiveness of our proposal is validated by 3 pretext task baselines and 4 different network backbones;
3. The proposal is flexible enough to be applied to other methods.

Requirements

This is my experimental environment when preparing this demo code.

• Ubuntu 18.04.4 LTS
• conda 4.8.4
• PyTorch 1.4.0

[Warning] We have met problems in other projects that different pytorch versions (1.7.0) might cause totally different results.

• python 3.8.3
• cuda 10.1
• accimage

Usage

Data preparation

I used resized RGB frames from this repo. Frames of videos in UCF101 and HMDB51 datasets can be downloaded directly without decoding.

Tips: There is a folder called TSP_Flows inside v_LongJump_j18_c03 folder in UCF101 dataset and you may meet a problem if you do not handle this. One solution is to delete this folder.

The folder architecture is like path/to/dataset/jpegs_256/video_id/frames.jpg.

Then, you need to edit datasets/ucf101.py and datasets/hmdb51.py to specify the path for dataset. Please change *_dataset_path in line #19.

Training self-supervised learning part with our PCL

python train_vcp_contrast.py

Default settings are

• Method: PCL (VCP)
• Backbone: R3D
• Modality: Res
• Augmentation: RandomCrop, ColorJitter, RandomGrayScale, GaussianBlur, RandomHorizontalFlip
• Dataset: UCF101
• Split: 1

These settings are also fixed for the following process, we do not need to specify --model=r3d --modality=res.

The training will take around 33 hours on one V100 based on our experimental environment.

Models will be saved to ./logs/exp_name. Here, exp_name is directly generated by its corresponding settings.

Evaluation using video retrieval

python retrieve_clips.py --ckpt=/path/to/ssl/best_model --dataset=ucf101

Fine-tuning on video recognition

python ft_classify.py --ckpt=/path/to/ssl/best_model --dataset=ucf101

The testing process will automatically run after training is done.

Results

Video retrieval

Our PCL outperform a set of methods by a large margin. Here we list results using Resnet-18-3D as network backbone. For more results, please refer to our paper.

Video recognition

The table lists recognition results on UCF101 and HMDB51 datasets. Other results are from corresponding paper. Because this is the most widely used metrics, we show results based on 4 different network backbones.

Methods in this table do not contain those using other data modalities such as sound and text.

Citation

If you find our work helpful for your research, please consider citing the paper

@article{tao2021pcl,
    title={Pretext-Contrastive Learning: Toward Good Practices in Self-supervised Video Representation Leaning},
    author={Tao, Li and Wang, Xueting and Yamasaki, Toshihiko},
    journal={arXiv preprint arXiv:2010.15464},
    year={2021}
}

If you find the residual input helpful for video-related tasks, please consider citing the paper

@article{tao2020rethinking,
  title={Rethinking Motion Representation: Residual Frames with 3D ConvNets for Better Action Recognition},
  author={Tao, Li and Wang, Xueting and Yamasaki, Toshihiko},
  journal={arXiv preprint arXiv:2001.05661},
  year={2020}
}

@inproceedings{tao2020motion,
  title={Motion Representation Using Residual Frames with 3D CNN},
  author={Tao, Li and Wang, Xueting and Yamasaki, Toshihiko},
  booktitle={2020 IEEE International Conference on Image Processing (ICIP)},
  pages={1786--1790},
  year={2020},
  organization={IEEE}
}

Acknowledgements

Part of this code is reused from IIC and VCOP.