Home

Awesome

Long-Term Feature Banks for Detailed Video Understanding

Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krähenbühl, Ross Girshick <br/> In CVPR 2019. [Paper] <br/> <br/>

<div align="center"> <img src="figs/lfb_concept_figure.jpg" width="800"> </img></div> <br/> This is a Caffe2 based implementation for our CVPR 2019 paper on Long-Term Feature Banks (LFB). LFB provides supportive information extracted over the entire span of a video, to augment state-of-the-art video models that otherwise would only view short clips of 2-5 seconds. Our experiments demonstrate that augmenting 3D CNNs with an LFB yields state-of-the-art results on AVA, EPIC-Kitchens, and Charades.

Data Preparation and Installation

Please see DATASET.md, INSTALL.md for instructions.

Training and Inference

Please see GETTING_STARTED.md for details.

Results

The following documents a collection of models trained with this repository. Links to the trained models and output log files are provided. The performance is evaluated on the validation set.

Note that all models here are not the original models used in paper, but reproduced by this code base. The reproduced performance reported here is very close to (or slightly better than) what's reported in paper, but not exactly the same due to the stochastic nature of training.

AVA

configbackbonemethodmAPmodel idmodel
ava_r50_baselineR50-I3D-NL3D CNN22.2102760666model
ava_r50_lfb_avgR50-I3D-NLLFB-Avg23.3103505104model, lfb model
ava_r50_lfb_maxR50-I3D-NLLFB-Max23.9103505159model, lfb model
ava_r50_lfb_nlR50-I3D-NLLFB-NL-2L25.8102824705model, lfb model
ava_r50_lfb_nl_3lR50-I3D-NLLFB-NL-3L25.9106403526model, lfb model
ava_r101_baselineR101-I3D-NL3D CNN23.2102760714model
ava_r101_lfb_nl_3lR101-I3D-NLLFB-NL-3L26.9 (multi-crop: 27.7)105206523model, lfb model

EPIC Kitchens Verb

configbackbonemethodtop1top5model idmodel
epic_verb_r50_baselineR50-I3D-NL3D CNN50.781.1103704809model
epic_verb_r50_lfb_avgR50-I3D-NLLFB-Avg52.982.5103777391model, lfb model
epic_verb_r50_lfb_maxR50-I3D-NLLFB-Max53.381.0103777432model, lfb model
epic_verb_r50_lfb_nlR50-I3D-NLLFB-NL52.381.8103777046model, lfb model

EPIC Kitchens Noun

configbackbonemethodtop1top5model idmodel
epic_noun_r50_baselineR50-I3D-NL3D CNN26.251.0104421642model
epic_noun_r50_lfb_avgR50-I3D-NLLFB-Avg29.156.3103875866model
epic_noun_r50_lfb_maxR50-I3D-NLLFB-Max32.056.5103875899model
epic_noun_r50_lfb_nlR50-I3D-NLLFB-NL29.555.4103706990model

EPIC Kitchens Action

configbackbonemethodtop1top5
epic_verb_r50_baseline & epic_noun_r50_baselineR50-I3D-NL3D CNN19.438.1
epic_verb_r50_lfb_avg & epic_noun_r50_lfb_avgR50-I3D-NLLFB-Avg21.241.3
epic_verb_r50_lfb_max & epic_noun_r50_lfb_maxR50-I3D-NLLFB-Max22.941.2
epic_verb_r50_lfb_nl & epic_noun_r50_lfb_nlR50-I3D-NLLFB-NL21.840.5

Note: To make action predictions, we combine a verb model and a noun model, as opposed to training a separate action model. Performance in this table is computed using the verb/noun models from the tables above. Please see GETTING_STARTED.md for instructions on how to do this.

Charades

configbackbonemethodmAPmodel idmodel
charades_r50_baselineR50-I3D-NL3D CNN38.3102766107model
charades_r50_lfb_avgR50-I3D-NLLFB-Avg38.4102999065model, lfb model
charades_r50_lfb_maxR50-I3D-NLLFB-Max38.6102999121model, lfb model
charades_r50_lfb_nlR50-I3D-NLLFB-NL40.3100866795model, lfb model
charades_r101_baselineR101-I3D-NL3D CNN40.4103560426model
charades_r101_lfb_avgR101-I3D-NLLFB-Avg40.8103676713model, lfb model
charades_r101_lfb_maxR101-I3D-NLLFB-Max41.0103676788model, lfb model
charades_r101_lfb_nlR101-I3D-NLLFB-NL42.5103641815model, lfb model

License

Video-long-term-feature-banks is Apache 2.0 licensed, as found in the LICENSE file.

Citation

@inproceedings{lfb2019,
  Author    = {Chao-Yuan Wu and Christoph Feichtenhofer and Haoqi Fan
               and Kaiming He and Philipp Kr\"{a}henb\"{u}hl and
               Ross Girshick},
  Title     = {{Long-Term Feature Banks for Detailed Video Understanding}},
  Booktitle = {{CVPR}},
  Year      = {2019}}