Awesome
BraVL
This is the official code for the paper "Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2023)" (https://ieeexplore.ieee.org/document/10089190).
- Authors: Changde Du, Kaicheng Fu, Jinpeng Li, Huiguang He
Preliminaries
This code was developed and tested with:
- Python version 3.7.0
- PyTorch version 1.9.0
- CUDA version 11.2
- The conda environment defined in
environment.yml
First, set up the conda enviroment as follows:
conda env create -f environment.yml # create conda env
conda activate BraVL # activate conda env
Download data
Second, download the pre-processed trimodal data from https://figshare.com/articles/dataset/BraVL/17024591, unzip them, and put them at "./data" directory:
unzip DIR-Wiki.zip -d BraVL_fMRI/data/
unzip GOD-Wiki.zip -d BraVL_fMRI/data/
unzip ThingsEEG-Text.zip -d BraVL_EEG/data/
Note that, the raw (image and brain fMRI/EEG) data are not included here because they are too large. Raw ImageNet images and brain fMRI data can be downloaded from the corresponding official site. We provide python scripts for feature extraction and data preprocessing.
Experiments
Experiments can be started by running the job_trimodal
script.
running BraVL on the Image-Text-fMRI datasets
cd BraVL_fMRI
bash job_trimodal
running BraVL on the Image-Text-EEG datasets
cd BraVL_EEG
bash job_trimodal
Cite
Please cite our paper if you use this code in your own work:
@article{du2023decoding,
title={Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features},
author={Du, Changde and Fu, Kaicheng and Li, Jinpeng and He, Huiguang},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
year={2023},
publisher={IEEE}
}
If you have any questions about the code or the paper, we are happy to help!