Awesome
vsumm-reinforce
This is the official implementation of the AAAI'18 paper Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward. The code is based on Theano (version 0.9.0
).
Pytorch implementation can be found here.
Preparation
To get the datasets and models, you will need wget
.
Run the following commands in order
git clone https://github.com/KaiyangZhou/vsumm-reinforce
cd vsumm-reinforce
# download datasets.tar.gz
wget http://www.eecs.qmul.ac.uk/~kz303/vsumm-reinforce/datasets.tar.gz
tar -xvzf datasets.tar.gz
# download models.tar.gz
wget http://www.eecs.qmul.ac.uk/~kz303/vsumm-reinforce/models.tar.gz
tar -xvzf models.tar.gz
Updates: The QMUL server is inaccessible. Download the datasets from this google drive link.
How to train
Training code is implemented in vsum_train.py
. To train a RNN, run
python vsum_train.py --dataset datasets/eccv16_dataset_tvsum_google_pool5.h5 --max-epochs 60 --hidden-dim 256
How to test
Test code is implemented in vsum_test.py
. For example, to test with our models, simply run
python vsum_test.py -model models/model_tvsum_reinforceRNN.h5 -d tvsum
python vsum_test.py -model models/model_tvsum_reinforceRNN_sup.h5 -d tvsum
python vsum_test.py -model models/model_summe_reinforceRNN.h5 -d summe
python vsum_test.py -model models/model_summe_reinforceRNN_sup.h5 -d summe
Output results are saved to log-test/results.h5
. To visualize score-vs-gtscore, you can use visualize_results.py
by
python visualize_results.py -p log-test/result.h5
Visualize summary
You can use summary2video.py
to transform the binary machine_summary
to real summary video. You need to have a directory containing video frames. The code will automatically write summary frames to a video where the frame rate can be controlled. Use the following command to generate a .mp4
video
python summary2video.py -p path_to/result.h5 -d path_to/video_frames -i 0 --fps 30 --save-dir log --save-name summary.mp4
Please remember to specify the naming format of your video frames on this line.
Citation
@article{zhou2017reinforcevsumm,
title={Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward},
author={Zhou, Kaiyang and Qiao, Yu and Xiang, Tao},
journal={arXiv:1801.00054},
year={2017}
}