Home

Awesome

PWC

PWC

PWC

All-in-one

Code for the paper: All in One: Exploring Unified Video-Language Pre-training Arxiv

ppl

News

Install

1. PytorchLighting

In this work, we use PytorchLighting for distributed training with mixed precision. Install pytorch and PytorchLighting first.

conda create -n allinone python=3.7
source activate allinone
cd [Path_To_This_Code]
pip install -r requirements.txt

If all packages include ffmpeg installed, please skip step 2.

2. On-the-fly decode (may skip)

To speed up the pre-training, we adopt on-the-fly decode for fast IO. Install ffmpeg as below.

1. ffmpeg

sudo conda install -y ffmpeg

Please install the required packages if not included in the requirements.txt.

If you server cannot connect to http or install ffmpeg slowly. Please download static binary file from FFmpeg Static Builds and then add to path variable, as follows:

export PATH=[PATH_TO_Dir/]ffmpeg-git-20220108-amd64-static:$PATH

2. pytorch video

Install pytorchvideo (for data augmentation) as below:

pip install ffmpeg-python
pip install pytorchvideo

Download Pretrained Weights

We provide three pretrained weights in google driver.

ModelPT DataParameterPretrained WeightTrained LogHparams
All-in-one-TiWebvid+HowTo12MGoogle DriverGoogle DriverGoogle Driver
All-in-one-SWebvid+HowTo33MGoogle DriverGoogle DriverGoogle Driver
All-in-one-BWebvid+HowTo110MGoogle DriverGoogle DriverGoogle Driver
All-in-one-B+Webvid+HowTo+<br>CC3110MGoogle DriverGoogle DriverGoogle Driver
All-in-one-B+Webvid+YTT+HowTo+<br>CC3+CC12+Coco+VG+SBU110MGoogle DriverGoogle DriverGoogle Driver

After downloaded these pretrained weights, move them into pretrained dir.

mkdir pretrained
cp *.ckpt pretrained/

Compare with state-of-the-arts

ModelParamDataFramesTGIF-ActionTGIF-FrameMSR R@5MSR R@10
ClipBERT137MI:Coco+VG8 x 282.959.449.263.5
VIOLET198MV:Webvid+<br>I:CC31687.1-63.073.4
All-in-one-S33MV:WebVid+Howto391.264.061.570.9
All-in-one-B110MV:WebVid+Howto392.964.267.077.1
All-in-one-B+110MV:Webvid+<br>I:CC3395.467.268.177.3
All-in-one-B+110MV:Webvid+YTT+HowTo+<br>I:CC3+CC12+Coco+VG+SBU396.368.570.379.2

I is short for Image and V is short for Video in this table.

Dataset Preparation

See DATA.md

Pre-training

Full Video Pre-training

See TRAIN.md

Co-training with Image Dataset (All-in-one+)

See COTRAIN.md

Evaluation on Downstream Tasks

See EVAL.md

By unified design and sparse sampling, AllInOne show much small flops.

Citation

If you find our work helps, please cite our paper.

@article{wang2022allinone,
  title={All in One: Exploring Unified Video-Language Pre-training},
  author={Wang, Alex Jinpeng and Ge, Yixiao and Yan, Rui and Ge Yuying and Lin, Xudong and Cai, Guanyu  and Wu, Jianping and Shan, Ying and Qie, Xiaohu and Shou, Mike Zheng},
  journal={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2023}
}

Contact

Email: awinyimgprocess at gmail dot com

If you have any problem or have difficult in reproducing the results reported in this code, you can email to me or open a question in issues. We are also willing to merge the code if transfer our All-in-one to different tasks or datasets.

Acknowledgement

This work is mainly based on ViLT, Frozen and Merlot.

License

MIT