Awesome
Event Sequence Generation Network
Unoffical re-implementation of Event Sequence Selection Network (ESGN) in paper titled "streamlined dense video captioning". Note that we do not adopt SST to encode the proposal-level features, which is different from the original model.
Environment
- Python 3.6.2
- CUDA 10.0, PyTorch 1.2.0 (may work on other versions but has not been tested)
- other modules, run
pip install -r requirement.txt
Prerequisites
-
C3D feature. Download C3D feature files (
sub_activitynet_v1-3.c3d.hdf5) from here. Convert the h5 file into npy files and place them into./data/c3d. -
Download annotation files and pre-generated proposals files (top100 proposals generated by DBG) from Google Drive, and place them into
./data.
Usage
- Training
cfg_path=cfgs/esgn.yml
python train.py --cfg_path $cfg_path
the checkpoint files are saved in this folder ./save.
- Validation
python eval.py --eval_folder esgn_c3d_run0
- Validation with re-ranking
python eval.py --eval_folder esgn_c3d_run0 --eval_esgn_rerank
Performance
| Model | proposal model | Avg proposal number | Avg Recall | Avg Precision | F1 | download |
|---|---|---|---|---|---|---|
| Original ESGN | SST | 2.85 | 55.58 | 57.57 | 56.66 | |
| My reimpl. | DBG | 2.73 | 52.67 | 58.90 | 55.62 | url |
| My reimpl. with reranking | DBG | 1.66 | 37.66 | 67.47 | 48.33 |
Pretrained model
Download the pre-trained model and put it into ./save/esgn_c3d_run0, then run python eval.py --eval_folder esgn_c3d_run0.
References
- Awesome ImageCaptioning.pytorch project.
- Official implementation of "Weakly Supervised Dense Event Captioning in Videos".