Home

Awesome

[ECCV2022] MorphMLP [arxiv]

Our MorphMLP paper was accepted to ECCV 2022!!

We current release the code and models for:

Update

Aug,3rd 2022

[Initial commits]:

  1. Pretrained models on Kinetics-400, Something-Something V1

Model Zoo

The ImageNet-1K pretrained models, followed models and logs can be downloaded on Google Drive: total_models.

We also release the models on Baidu Cloud: total_models (bbyy).

Note

Kinetics-400

Model#FrameSampling StrideFLOPsTop1ModelLogconfig
MorphMLP-S16x1x44268G78.7googlegoogleconfig
MorphMLP-S32x1x44532G79.7googlegoogleconfig
MorphMLP-B16x1x44392G79.5googlegoogleconfig
MorphMLP-B32x1x44788G80.8googlegoogleconfig

Something-Something V1

ModelPretrain#FrameFLOPsTop1ModelLogconfig
MorphMLP-SIN-1K16x1x167G50.6[soon][soon]config
MorphMLP-SIN-1K16x3x1201G53.9[soon][soon]config
MorphMLP-BIN-1K16x3x1294G55.1googlegoogleconfig
MorphMLP-BIN-1K32x3x1591G57.4googlegoogleconfig

Something-Something V2

ModelPretrain#FrameFLOPsTop1ModelLogconfig
MorphMLP-SIN-1K16x3x1201G67.1[soon][soon]config
MorphMLP-SIN-1K32x3x1405G68.3[soon][soon]config
MorphMLP-BIN-1K16x3x1294G67.6[soon][soon]config
MorphMLP-BIN-1K32x3x1591G70.1[soon][soon]config

Usage

Installation

Please follow the installation instructions in INSTALL.md. You may follow the instructions in DATASET.md to prepare the datasets.

Training

  1. Download the pretrained models into the pretrained folder.

  2. Simply run the training code as followed:

python3 tools/run_net.py --cfg configs/K400/K400_MLP_S16x4.yaml DATA.PATH_PREFIX path_to_data OUTPUT_DIR your_save_path

[Note]:

Testing

We provide testing example as followed:

Kinetics400

python3 tools/run_net.py --cfg configs/K400/K400_MLP_S16x4.yaml DATA.PATH_PREFIX path_to_data TRAIN.ENABLE False  TEST.NUM_ENSEMBLE_VIEWS 4 TEST.NUM_SPATIAL_CROPS 1 TEST.CHECKPOINT_FILE_PATH your_model_path OUTPUT_DIR your_output_dir

SomethingV1&V2

python3 tools/run_net.py   --cfg configs/SSV1/SSV1_MLP_B32.yaml DATA.PATH_PREFIX your_data_path TEST.NUM_ENSEMBLE_VIEWS 1 TEST.NUM_SPATIAL_CROPS 3 TEST.CHECKPOINT_FILE_PATH your_model_path OUTPUT_DIR your_output_dir

Specifically, we need to set the number of crops&clips and your checkpoint path then run multi-crop/multi-clip test:

Set the number of crops and clips:

Multi-clip testing for Kinetics

TEST.NUM_ENSEMBLE_VIEWS 4
TEST.NUM_SPATIAL_CROPS 1

Multi-crop testing for Something-Something

TEST.NUM_ENSEMBLE_VIEWS 1
TEST.NUM_SPATIAL_CROPS 3

You can also set the checkpoint path via:

TEST.CHECKPOINT_FILE_PATH your_model_path

Cite MorphMLP

If you find this repository useful, please use the following BibTeX entry for citation.

@article{zhang2021morphmlp,
  title={Morphmlp: A self-attention free, mlp-like backbone for image and video},
  author={Zhang, David Junhao and Li, Kunchang and Chen, Yunpeng and Wang, Yali and Chandra, Shashwat and Qiao, Yu and Liu, Luoqi and Shou, Mike Zheng},
  journal={arXiv preprint arXiv:2111.12527},
  year={2021}
}

Acknowledgement

This repository is built based on SlowFast and Uniformer repository.