Awesome
DDM-Net (CVPR 2022)
This repo holds the codes of paper: "Progressive Attention on Multi-Level Dense Difference Maps for Generic Event Boundary Detection", accepted in CVPR 2022.
News
[2022.5.8] The code is available now.
[2022.3.3] DDM-Net is accepted to CVPR 2022.
[2021.11.16] Our DDM-Net ranks 1st on the leaderboard of LOVEU@CVPR 2021, outperforming the top1 solution of LOVEU Challenge 2021.
Overview
This paper presents a modular framework for the task of generic event boundary detection (GEBD). To perceive diverse temporal variations and learn complex semantics of generic event boundaries, our method progressively attends to multi-level dense difference maps (DDM). Thanks to holistic temporal modeling and joint feature learning across modalities, our DDM-Net outperforms the previous state-of-the-art methods by a large margin on Kinetics-GEBD and TAPOS benchmark. In addition, our method is better than winner solutions of LOVEU Challenge@CVPR 2021, further demonstrating the efficacy of DDM-Net.
Dependencies
Python 3.7 or higher
PyTorch 1.6 or higher
einops
ipdb
Guide
Please refer to GUIDE for preparing input data and generating boundary predictions.
Performance
Dataset | F1@0.05 | F1@0.25 | F1@0.5 | Avg F1 | checkpoint | pickle |
---|---|---|---|---|---|---|
Kinetics-GEBD | 76.43% | 88.70% | 90.16% | 87.26% | ckpt | pkl |
Training
Use tools/train.sh
to train DDM-Net.
python DDM-Net/train.py \
--dataset kinetics_multiframes \
--train-split train \
--val-split val \
--num-classes 2 \
--batch-size 16 \
--n-sample-classes 2 \
--n-samples 16 \
--lr 0.00001 \
--warmup-epochs 0 \
--epochs 5 \
--decay-epochs 2 \
--model multiframes_resnet \
--pin-memory \
--sync-bn \
--amp \
--native-amp \
--distributed \
--eval-metric loss \
--log-interval 50 \
--port 16580 \
--eval-freq 1
Testing
Inference with tools/test.sh
.
python DDM-Net/test.py \
--dataset kinetics_multiframes \
--val-split val \
-b 128 \
--resume checkpoint.pth.tar
Citation
If you find DDM-Net useful in your research, please cite us using the following entry:
@InProceedings{Tang_2022_CVPR,
author = {Tang, Jiaqi and Liu, Zhaoyang and Qian, Chen and Wu, Wayne and Wang, Limin},
title = {Progressive Attention on Multi-Level Dense Difference Maps for Generic Event Boundary Detection},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2022},
pages = {3355-3364}
}
Acknowledgement
We especially thank the contributors of the GEBD, RepNet, TSM and DETR for providing helpful code.
Thanks to Fengyuan Shi and Xun Jiang for their help.
Contact
Jiaqi Tang: jqtang@smail.nju.edu.cn