Home

Awesome

Learning Factorized Multimodal Representations

Pytorch implementation for learning factorized multimodal representations using deep generative models.

Correspondence to:

Paper

Learning Factorized Multimodal Representations<br> Yao-Hung Hubert Tsai*, Paul Pu Liang*, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov<br> ICLR 2019. (*equal contribution)

Installation

First check that the requirements are satisfied:</br> Python 3.6/3.7</br> PyTorch 0.4.0</br> numpy 1.13.3</br> sklearn 0.20.0

The next step is to clone the repository:

git clone https://github.com/pliang279/factorized.git

Dataset

Please download the latest version of the CMU-MOSI, CMU-MOSEI, POM, and IEMOCAP datasets which can be found at https://github.com/A2Zadeh/CMU-MultimodalSDK/

Scripts

Please run

python mfm_test_mosi.py

in the command line.

Similar commands for loading and running models for other datasets can be found in mfm_test_mmmo.py, mfm_test_moud.py etc.

If you use this code, please cite our paper:

@inproceedings{DBLP:journals/corr/abs-1806-06176,
  title     = {Learning Factorized Multimodal Representations},
  author    = {Yao{-}Hung Hubert Tsai and
               Paul Pu Liang and
               Amir Zadeh and
               Louis{-}Philippe Morency and
               Ruslan Salakhutdinov},
  booktitle={ICLR},
  year={2019}
}

Related papers and repositories building upon these datasets:</br> CMU-MOSEI dataset: paper, code</br> Memory Fusion Network: paper, code</br> Multi-Attention Recurrent Network: paper, code</br> Graph-MFN: paper, code</br> Multimodal Transformer: paper, code</br> Multimodal Cyclic Translations: paper, code