Home

Awesome

<div align="center"> <h2>UMBRAE: Unified Multimodal Brain Decoding (ECCV 2024)</h2> <div> <a href='https://weihaox.github.io/' target='_blank'>Weihao Xia</a><sup>1</sup>&emsp; <a href='https://team.inria.fr/rits/membres/raoul-de-charette/' target='_blank'>Raoul de Charette</a><sup>2</sup>&emsp; <a href='https://www.cl.cam.ac.uk/~aco41/' target='_blank'>Cengiz Öztireli</a><sup>3</sup>&emsp; <a href='http://www.homepages.ucl.ac.uk/~ucakjxu/' target='_blank'>Jing-Hao Xue</a><sup>1</sup>&emsp; </div> <div> <sup>1</sup>University College London&emsp; <sup>2</sup>Inria&emsp; <sup>3</sup>University of Cambridge&emsp; </div> <br/> <!-- <h3 align="center">Preprint</h3> --> <!-- --- --> <!-- <h4 align="center"> <a href="https://weihaox.github.io/BrainX" target='_blank'>[Project Page]</a> • <a href="https://www.youtube.com/watch?v=cUdkeigISOo" target='_blank'>[Video]</a> • <a href="https://arxiv.org/pdf/2310.02265" target='_blank'>[arXiv]</a> <br> <br> <img src="https://visitor-badge.laobi.icu/badge?page_id=weihaox/BrainX" width="8%" alt="visitor badge"/> </h4> -->

<a href="https://pytorch.org/get-started/locally/"><img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-ee4c2c?logo=pytorch&logoColor=white"></a> Paper Hugging Face Open In Colab

</div> <div align="center"><tr> <img src="docs/images/teaser.png" width="100%"/> </tr></div> <div align="center"><tr> <img src="docs/images/demo_example.png" width="100%"/> </tr></div> <br/> <p>UMBRAE decodes multimodal explanations from brain signals. (1) We introduce a <b>universal brain encoder</b> for multimodal-brain alignment and recover conceptual and spatial details by using multimodal large language models. (2) We introduce <b>cross-subject training</b> to overcome unique brain patterns of different individuals. This allows brain signals from multiple subjects to be trained within the same model. (3) Our method supports <b>weakly-supervised subject adaptation</b>, enabling the training of a model for a new subject in a data-efficient manner. (4) For evaluation, we introduce <b>BrainHub</b>, a brain understanding benchmark, based on NSD and COCO.

News :triangular_flag_on_post:

Method

Overview of UMBRAE. Our brain encoder includes subject-specific tokenizers and a universal perceive encoder. Brain signals from multiple subjects are mapped into a common feature space, enabling cross-subject training and weakly-supervised subject adaptation. The brain encoder learns to align neural signals with image features. During inference, the learned encoder interacts with MLLMs and performs brain understanding tasks according to given prompts.

<div align="center"><tr> <img src="docs/images/overview.png" width="90%"/> </tr></div>

Installation

Environment

conda create -n brainx python=3.10
conda activate brainx
pip install -r requirements.txt

Download Data and Checkpoints

The training and inference scripts support automatically downloading the dataset if the designated path is empty. However, this process can be quite slow. You can try the following script to download all data in advance if this happens. Please fill out the NSD Data Access form and agree to the Terms and Conditions.

Download Checkpoints from Hugging Face.

bash download_data.sh
bash download_checkpoint.sh

Inference

Our method inherits multimodal understanding capabilities of MLLMs, enabling the switch between different tasks through different prompts. You can either use the prompts listed in our paper or create customised instructions according to actual needs. Please specify brainx-v-1-4 or brainx.

exp='brainx-v-1-4' # 'brainx'

prompt_caption='Describe this image <image> as simply as possible.'

for sub in 1 2 5 7
do
python inference.py --data_path 'nsd_data' --fmri_encoder 'brainx' --subj $sub \
    --prompt "$prompt_caption" --brainx_path "train_logs/${exp}/last.pth" \
    --save_path "evaluation/eval_caption/${exp}"
done

Given that identified classes might be named differently, or simply absent from ground truth labels, we evaluate bounding boxes through REC. We use prompt "Locate <expr> in <image> and provide its coordinates, please", but others like "Can you point out <expr> in the image and provide the bounding boxes of its location?" shall also work.

for sub in 1 2 5 7
do
    python inference_rec.py --data_path 'nsd_data' --fmri_encoder 'brainx' \
      --subj $sub --brainx_path "train_logs/${exp}/last.pth" \
      --save_path "evaluation/eval_bbox_rec/${exp}/sub0${sub}_dim1024"
done

Training

Single-Subject Training

accelerate launch --num_processes=1 --num_machines=1 --gpu_ids='0' train.py \
    --data_path 'nsd_data' --fmri_encoder 'brainxs' --subj 1 \
    --model_save_path 'train_logs/demo_single_subject/sub01_dim1024'

Cross-Subject Training

accelerate launch --num_processes=1 --num_machines=1 --gpu_ids='0' train_brainx.py \
    --data_path 'nsd_data' --fmri_encoder 'brainx' --batch_size 128 --num_epochs 300 \
    --model_save_path 'train_logs/demo_cross_subject' --subj 1 2 5 7

Weakly-Supervised Subject Adaptation

If you would like to adapt to a new subject, for example, S7, first train a model with other available subjects (S1, S2, S5) using the above cross-subject training. Then train the new subject using the following command.

sub=7
data_ratio=1.0
accelerate launch --num_processes=1 --num_machines=1 --gpu_ids='0' train_brainx_adaptation.py \
    --data_path 'nsd_data' --fmri_encoder 'brainxc' --batch_size 128 --num_epochs 240 \
    --subj $sub --data_ratio $data_ratio \
    --encoder_path 'train_logs/demo_cross_subject/brainx_adaptation_125/last.pth' \
    --model_save_path "train_logs/demo_weak_adaptation/brainx_adaptation_${sub}_${data_ratio}"

Evaluation

The benchmark, including groundtruth data, evaluation scripts, and baseline results, is in brainhub.

  1. Download brainhub to the root path: git clone https://github.com/weihaox/BrainHub
  2. Process groundtruth test images: python processing/decode_images.py
  3. Run evaluation for brain captioning and grounding:
cd BrainHub
for sub in 1 2 5 7
do
    python eval_caption.py ../umbrae/evaluation/eval_caption/${exp}/sub0${sub}_dim1024/fmricap.json \
        caption/images --references_json caption/fmri_cococap.json
    python eval_bbox_rec.py --path_out "../umbrae/evaluation/eval_bbox_rec/${exp}/sub0${sub}_dim1024"
done

We also provide baseline results associated with BrainHub, including the captioning results from SDRecon, BrainCap, and OneLLM, as well as the captioning and grounding results from UMBRAE.

TODO

Acknowledgements

We thank the authors of SDRecon, BrainCap, and OneLLM for providing the codes or the results. We also express gratitude for NSD and COCO, which were used to construct our brainhub. The training script is based on MindEye. We utilize the pretrained models Shikra and LLaVA as the MLLMs. Thanks for the awesome research works.

Citation

@inproceedings{xia2024umbrae,
  author    = {Xia, Weihao and de Charette, Raoul and Öztireli, Cengiz and Xue, Jing-Hao},
  title     = {UMBRAE: Unified Multimodal Brain Decoding},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2024},
}