Awesome
Learning to Cut by Watching Movies
Official Code of ICCV 2021 Paper: Learning to Cut by Watching Movies
[ ArXiv | Project Website | ICCV2021 ]
Learning to Cut by Watching Movies. Alejandro Pardo*, Fabian Caba Heilbron, Juan León Alcázar, Ali Thabet, Bernard Ghanem. In ICCV, 2021.
<div align="center" valign="middle"><img height="450px" src="./pull_figure.jpg"></div>Installation
Clone the repository and move to folder:
git clone https://github.com/PardoAlejo/LearningToCut.git
cd LearningToCut
Install environmnet:
conda env create -f ltc-env.yml
Data
Download the following resources and extract the content in the appropriate destination folder. See table.
| Resource | Drive File | Destination Folder |
|---|---|---|
| Train Annotations | link | ./data/ |
| Val Annotations | link | ./data/ |
| Video Durations | link | ./data/ |
| Video Features | link | ./data/ |
| Audio Features | link | ./data/ |
| Best Model | link | ./checkpoints/ |
If you want to extract features yourself, or you need the original videos instead, please refer to data/DATA.md
The folder structure should be as follows:
README.md
ltc-env.yml
│
├── data
│ ├── ResNexT-101_3D_video_features.h5
│ ├── ResNet-18_audio_features.h5
│ ├── subset_moviescenes_shotcuts_train.csv
│ ├── subset_moviescenes_shotcuts_val.csv
│ └── durations.csv
│
├── checkpoints
| ├── best_state.ckpt
│
└── scripts
Inference
Copy paste the following commands in the terminal. </br>
Load environment:
conda activate ltc
cd scripts/
Inference on val set
sh inference.sh
Expected results (Table 1 of the Paper):
| Method | AR1-D1 | AR3-D1 | AR5-D1 | AR10-D1 | AR1-D2 | AR3-D2 | AR5-D2 | AR10-D2 | AR1-D3 | AR3-D3 | AR5-D3 | AR10-D3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Random | 0.64% | 1.91% | 3.15% | 6.28% | 1.85% | 5.65% | 9.32% | 18.52% | 3.67% | 10.67% | 17.62% | 33.91% |
| Raw | 1.16% | 3.97% | 6.36% | 11.72% | 2.51% | 8.32% | 13.15% | 24.25% | 3.73% | 12.19% | 19.33% | 34.97% |
| LTC | 8.18% | 17.95% | 24.44% | 30.35% | 15.30% | 35.11% | 48.26% | 59.42% | 19.18% | 46.32% | 64.30% | 79.35% |
Cite us
@InProceedings{Pardo_2021_ICCV,
author = {Pardo, Alejandro and Caba, Fabian and Alcazar, Juan Leon and Thabet, Ali K. and Ghanem, Bernard},
title = {Learning To Cut by Watching Movies},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2021},
pages = {6858-6868}
}