Awesome

Multi-Modal Multi-Action Video Recognition

We release the code of the "Multi-Modal Multi-Action Video Recognition" framework proposed in our ICCV2021 paper and the extended work.

The main idea of the framework is to explore multi-action relations via utilizing multi-modal information in videos. We adopt GCN (Graph Convolutional Network) and Transformer Network as the relation learners, and also realize a multi-modal joint learning strategy for multi-action video recognition. The core implementation for the framework are lib/models/m3a_helper.py, lib/models/m3a_relation_learner.py, and lib/models/m3a_net_builder.py.

Preparation

Please follow the instructions in PREPARATION.md for installation and data preparation.

Get Started

Please refer to GETTING_STARTED.md to run the framework.

Model Zoo

The models and results are provided in MODEL_ZOO.md.

License

Please see the LICENSE file for more details.

Acknowledgement

We really appreciate the contributors of following codebases.

• facebookresearch/SlowFast
• facebookresearch/VMZ
• kenshohara/3D-ResNets-PyTorch
• tkipf/pygcn
• Megvii-Nanjing/ML-GCN
• jadore801120/attention-is-all-you-need-pytorch

Citation

@inproceedings{shi2021multi,
  title={Multi-Modal Multi-Action Video Recognition},
  author={Shi, Zhensheng and Liang, Ju and Li, Qianqian and Zheng, Haiyong and Gu, Zhaorui and Dong, Junyu and Zheng, Bing},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={13678--13687},
  year={2021}
}