Awesome

<img src="figs/bear.png" width="30"/> BEAR: a new BEnchmark on video Action Recognition

This repo contains the data and pre-trained models in "A Large-scale Study of Spatiotemporal Representation Learning with a New Benchmark on Action Recognition"

Andong Deng*, Taojiannan Yang*, Chen Chen<br> Center for Research in Computer Vision, University of Central Florida

[CVF]

If you find our work useful in your research, please cite:

@article{deng2023BEAR,
  title={A Large-scale Study of Spatiotemporal Representation Learning with a New Benchmark on Action Recognition},
  author={Deng, Andong and Yang, Taojiannan and Chen, Chen},
  journal={arXiv preprint arXiv:2303.13505},
  year={2023}
}

Updates

04/21/2024 Update HuggingFace link for pre-trained models.

08/08/2023 Update Dropbox link for pre-trained models.

07/17/2023 BEAR is accepted by ICCV 2023!

03/24/2023 Update Dropbox link for Mini-Sports1M.

03/23/2023 Initial commits

Introduction

To comprehensively probe the effectiveness of spatiotemporal representation learning, we introduce <img src="figs/bear.png" width="14"/>BEAR, a new BEnchmark on video Action Recognition. <img src="figs/bear.png" width="14"/>BEAR is a collection of 18 video datasets grouped into 5 categories (anomaly, gesture, daily, sports, and instructional), which covers a diverse set of real-world applications. With <img src="figs/bear.png" width="14"/>BEAR, we thoroughly evaluate 6 common spatiotemporal models pre-trained by both supervised and self-supervised learning. We also report transfer performance via standard finetuning, few-shot finetuning, and unsupervised domain adaptation. Our observation suggests that current state-of-the-arts cannot solidly guarantee high performance on datasets close to real-world applications and we hope <img src="figs/bear.png" width="14"/>BEAR can serve as a fair and challenging evaluation benchmark to gain insights on building next-generation spatiotemporal learners.

The evaluation is extremely simple since we provide all scripts in this codebase. The users only need to download datasets and run the scripts provided.

Datasets

The following table includes all the statistics about the 18 datasets collected in <img src="figs/bear.png" width="14"/>BEAR:

Datasets Download and Pre-processing

We provide downloading and pre-processing pipeline here for each dataset.

The HuggingFace link for part of BEAR datasets are here:

Mini-Sports1M Jester FineGym MOD20 MPII-Cooking2

Pre-trained Models

We prepare Kinetics-400 pre-trained models with both supervised and self-supervised pre-training:

The updated HuggingFace Link for both self-supervised pretraining and supervised pretraining are here:

SSL SUP

The pre-trained models can be downloaded from below if needed:

Benchmark

Based on pre-trained models on Kinetics400, we provide 4 types of evaluation paradigms in <img src="figs/bear.png" width="14"/>BEAR:

BEAR-Standard<br> BEAR-Fewshot<br> BEAR-Zeroshot<br> BEAR-UDA<br>

Standard Finetuning

We build our stanard finetuning based on a popular video understanding toolbox MMAction2.

We provide specific training steps here.

The finetuning results of supervised pre-training are shown below:

The finetuning results of self-supervised pre-training are shown below:

Few-shot Finetuning

Please follow the instructions here to perform few-shot evaluation on <img src="figs/bear.png" width="14"/>BEAR.

The few-shot results are shown below:

Zero-shot Evaluation

We build our zero-shot part based on the popular CLIP and ActionCLIP. Follow the instructions here to evaluate zero-shot performance on <img src="figs/bear.png" width="14"/>BEAR.

The zero-shot results are shown below:

Domain Adaptation

Please follow the instructions here to perform UDA evaluation on <img src="figs/bear.png" width="14"/>BEAR.

The UDA baseline results are shown below: