Awesome
Frame Flexible Network (CVPR2023)
<div align="left"> <a><img src="fig/smile.png" height="70px" ></a> <a><img src="fig/neu.png" height="70px" ></a> <a><img src="fig/uva.png" height="70px" ></a> </div>arXiv | Primary contact: Yitian Zhang
@article{zhang2023frame,
title={Frame Flexible Network},
author={Zhang, Yitian and Bai, Yue and Liu, Chang and Wang, Huan and Li, Sheng and Fu, Yun},
journal={arXiv preprint arXiv:2303.14817},
year={2023}
}
TL,DR
Motivation: Existing video recognition algorithms always conduct separated training pipelines for inputs with different frame numbers, which requires repetitive training operations and multiplying storage costs.Observation: If we evaluate the model at frames which are not used in training, the performance will drop significantly (see Figure above), which is summarized as Temporal Frequency Deviation phenomenon.Solution: We propose a general framework, named Frame Flexible Network (FFN), which not only enables the model to be evaluated at different frames to adjust its computation, but also reduces the memory costs of storing multiple models significantly.Strength: (1) One-shot Training (2) Obvious Performance Gain (3) Significant Parameter Saving (4) Flexible Computation Adjustment (5) Easy Installation and Strong Compatibility.
Datasets
Please follow the instruction of TSM to prepare the Something-Something V1/V2, Kinetics400, HMDB51 datasets.
Support Models
FFN is a general framework and can be easily applied to existing methods for stronger performance and higher flexibility during inference. Currently, FFN supports the implementation of 2D Network: TSM, TEA; 3D Network: SlowFast; Transformer Network: Uniformer. Please feel free to contact us if you want to contribute the implementation of other methods.
Result
- Validation Across Architectures
FFN can obviously outperform Separated Training (ST) at all frames on different architectures with significant less parameters on Something-Something V1 dataset.
Here we provide the pretrained models on all these architectures:
| Model | Acc1.($v_{L}$) | Acc1.($v_{M}$) | Acc1.($v_{H}$) | Weight |
|---|---|---|---|---|
| TSM | 20.60% | 37.36% | 48.55% | link |
| TSM-ST | 39.71% | 45.63% | 48.55% | - |
| TSM-FFN | 42.85% | 48.20% | 50.79% | link |
| TEA | 21.78% | 41.49% | 51.23% | link |
| TEA-ST | 41.36% | 48.37% | 51.23% | - |
| TEA-FFN | 44.97% | 51.61% | 54.04% | link |
| SlowFast | 15.08% | 35.08% | 45.88% | link |
| SlowFast-ST | 39.91% | 44.12% | 45.88% | - |
| SlowFast-FFN | 43.90% | 47.11% | 47.27% | link |
| Uniformer | 22.38% | 47.98% | 56.71% | link |
| Uniformer-ST | 44.33% | 51.49% | 56.71% | - |
| Uniformer-FFN | 51.41% | 56.64% | 58.88% | link |
- Validation Across Datasets
FFN can obviously outperform Separated Training (ST) at all frames on different datasets with significant less parameters.
Here we provide the pretrained models on Something-Something V2:
| Model | Parameters | Acc1.($v_{L}$) | Acc1.($v_{M}$) | Acc1.($v_{H}$) | Weight |
|---|---|---|---|---|---|
| TSM | 25.6M | 31.52% | 51.55% | 61.02% | link |
| TSM-ST | 25.6x3M | 53.38% | 59.29% | 61.02% | - |
| TSM-FFN | 25.7M | 56.07% | 61.86% | 63.61% | link |
and Kinetics400:
| Model | Parameters | Acc1.($v_{L}$) | Acc1.($v_{M}$) | Acc1.($v_{H}$) | Weight |
|---|---|---|---|---|---|
| TSM | 25.6M | 64.10% | 69.77% | 73.16% | link |
| TSM-ST | 25.6x3M | 66.25% | 70.38% | 73.16% | - |
| TSM-FFN | 25.7M | 68.96% | 72.33% | 74.35% | link |
- Inference At Any Frame
FFN can be evaluated at any frame and outperform Seperated Training (ST) even at frames which are not used in training.
Get Started
We provide a comprehensive codebase for video recognition which contains the implementation of 2D Network, 3D Network and Transformer Network. Please go to the folders for specific docs.
Acknowledgment
Our codebase is heavily build upon TSM, SlowFast and Uniformer. We gratefully thank the authors for their wonderful works. The README file format is heavily based on the GitHub repos of my colleague Huan Wang, Xu Ma and Yizhou Wang. Great thanks to them! We also greatly thank the anounymous CVPR'23 reviewers for the constructive comments to help us improve the paper.