Awesome
pytorch-CTVSUM
This repository contains the official Pytorch implementation of the paper
Contrastive Losses Are Natural Criteria for Unsupervised Video Summarization
Zongshang Pang, Yuta Nakashima, Mayu Otani, Hajime Nagahara
In WACV2023
Installation
git clone https://github.com/pangzss/pytorch-CTVSUM.git
cd pytorch-CTVSUM
conda env create -f environment.yml
conda activate ctvsum
Dataset preparation
We use three datasets in our paper: TVSum, SumMe, and a random subset of Youtube8M.
TVSum and SumMe are used for training and evaluation, and Youtube8M is only used for training.
To prepare the datasets,
- Download raw videos from TVSum and SumMe and put them in ./data/raw
- Download the extracted features from GoogleDrive (GoogLeNet features for TVSum and SumMe, kindly provided by the authors of DRDSN, and quantized Inception features for Youtube8M).
- Put eccv_* files in ./data/interim, and unzip selected_features.zip in ./data/interim/youtube8M/
Training and Evaluation
Evaluation with only pretrained features
- In ./configs/aln_unif_config.yml, modify the dataset and evaluation settings, e.g.
data:
name: tvsum # summe
setting: Canonical # Augmented/Transfer
- Set
is_raw: True
- Set use Global Consistency or not
use_unif: True # False
- For Youtube8M features (quantized Inception), in ./configs/aln_unif_y8_config.yml, set
is_raw: True
hparams:
use_unif: True # False
- Run
./run_ablation.sh
./run_y8.sh
Contrastive refinement and evaluation
- For TVSum and SumMe, set training/evaluation setting in ./configs/aln_unif_config.yml, and decide whether to use global consistency or uniqueness filter
is_raw: False
use_unif: True # False
use_unq: True # False
The code will run 5-fold cross validation by default.
- For Youtube8M, similarly in ./configs/aln_unif_y8_config.yml,
is_raw: False
hparams:
use_unif: True # False
use_unq: True # False
- Run
./run_ablation.sh
./run_y8.sh
Acknowledgement
We would like to thank DRDSN, which provides the extracted features and the evaluation code for TVSum and SumMe. Moreover, we are thankful to the insightful work Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, which inspired our work.
Citation
@inproceedings{pang2023contrastive,
title={Contrastive Losses Are Natural Criteria for Unsupervised Video Summarization},
author={Pang, Zongshang and Nakashima, Yuta and Otani, Mayu and Nagahara, Hajime},
booktitle={WACV},
year={2023}
}