Awesome

<br></br>

UnIVAL <i>is a 0.25B-parameter unified model that is multitask pretrained on image and video-text data and target image, video and audio-text downstream tasks.</i>

<br></br>

Online Demos

Check out our demo on Huggingface Spaces: Spaces

General means the pretrained model before finetuning.

To easily play with our model we also provide several notebooks: VG.ipynb, VQA.ipynb, Captioning.ipynb, Video_Captioning.ipynb, and Audio_Captioning.ipynb

<br></br>

News

• [2023.12]: paper is accepted at TMLR!
• [2023.8.12]: we provide the scripts to train UnIVAL for audio/video-text tasks.
• [2023.7.31]: we provide here more details to reproduce the results with UnIVAL on Visual Grounding used in our Rewarded soups work.
• [2023.7.31]: Released of UnIVAL code and model weights! We will release the scripts to train and evaluate audio/video tasks later.

<br></br>

Table of Content

• Quantitative Results
• Installation
• Datasets and Checkpoints
• Training and Inference
• Zero-shot Evaluation
• Parameter Efficient Finetuning (PEFT): Training only the linear layer
• Multimodal Model Merging/Weight Interpolation
• Qualitative results
• Citation
• Acknowledgment

<br></br>

Results

Here are some results on several multimodal tasks.

Installation

Requirements

• python 3.7.4
• pytorch 1.13+
• torchvision 0.14.1+
• JAVA 1.8 (for COCO evaluation)

We recommend to first install pytorch before other libraries:

git clone https://github.com/mshukor/UnIVAL.git
pip install -r requirements.txt

Download the following model for captioning evaluation:

python -c "from pycocoevalcap.spice.spice import Spice; tmp = Spice()"

<br></br>

Datasets and Checkpoints

See datasets.md and checkpoints.md. <br></br>

Training and Inference

The scripts to launch pretraining, finetuning and evaluation can be found in run_scripts/ folder. Below we provide more details. The data are stored in .tsv files with different format depending on the training task. To restore training you need to provide the last checkpoint <code>checkpoint_last.pt</code> to <code>--restore-file</code>, and pass <code>--reset-dataloader --reset-meters --reset-optimizer</code> as argument.

We use slurm to launch the training/evaluation.

Image Processing

In some datasets, the images are encoded to base64 strings. To do this transformation you can use the following code:

from PIL import Image
from io import BytesIO
import base64

img = Image.open(file_name) # path to file
img_buffer = BytesIO()
img.save(img_buffer, format=img.format)
byte_data = img_buffer.getvalue()
base64_str = base64.b64encode(byte_data) # bytes
base64_str = base64_str.decode("utf-8") # str

Pretraining

Image Captioning

Visual Question Answering

Visual Grounding

Visual Entailment

Text-to-Image Generation

Zero-shot Evaluation

Here we provide the scripts for zero-shot evaluation on image-text tasks. You need to specify the path to pretrained model in each of these scripts:

• Image Caption on Nocaps: <code>caption/eval/eval_nocaps.sh</code>
• VQA on VizWiz: <code>vqa/eval/eval_vizwiz.sh</code>
• VQA on Nocaps: <code>vqa/eval/eval_okvqa.sh</code>

Parameter Efficient Finetuning

Training only the linear connection

Following eP-ALM, we experiment with efficient finetuning by training only the linear connection between the modality spcific-encoders and the language model, while keeping all other parameters frozen:

• Image Caption on COCO: <code>caption/onlylinear/unival_caption_stage_s2_onlylinear.sh</code>
• Video Caption on MSRVTT: <code>caption/onlylinear/unival_video_caption_stage_s2_onlylinear.sh</code>
• Audio Caption on Audiocaps: <code>caption/onlylinear/unival_audio_caption_stage_s2_onlylinear.sh</code>
• VQA on VQAv2: <code>vqa/onlylinear/unival_vqa_s2_onlylinear.sh</code>
• Video QA on MSRVTT: <code>vqa/onlylinear/unival_video_vqa_s2_onlylinear.sh</code>

To finetune the stage-1 pretrained model, you can use the scripts with s1.

Multimodal Model Merging

In this section we provide the details to reproduce the experiments for weight interpolation and different weight averaging experiments. The objective is to leverage the synergy between models finetuned on different multimodal tasks.

Weight interpolation

To average several models, you can use preprocess/average_save_models.py. There is two options, either you average many models with uniform interpolation coefficient, or you interpolate between 2 models with interpolation coefficient from 0 to 1. However, you can also customise this script as you like.

Once you saved the interpolated weights, you can use the following scripts to evaluate the model:

## image-text tasks
sh caption/eval/eval_caption_avg.sh
sh refcoco/eval/eval_refcocoplus_avg.sh
sh snli_ve/eval/eval_snli_ve_avg.sh
sh vqa/eval/eval_vqa_avg.sh

## video-text tasks 
sh vqa/eval/video/eval_video_qa_avg.sh
sh caption/eval/video/eval_msrvtt_video_caption_avg.sh

Ratatouille Finetuning

For Ratatouille finetuning, each one of the auxiliary models (e.g. models finetuned for captioning, vqa, visual grounding and visual entailment) are re-finetuned on the target task. At the end all obtained models are uniformly averaged.

The scripts to launch the finetuning and evaluation are in averaging/ratatouille/. You need also to use the weight averaging script in preprocess/average_save_models.py.

Fusing Finetuning

For Fusing finetuning, first the auxiliary models are averaged, then finetuned on the target task.

The scripts to launch the finetuning and evaluation are in averaging/fusing/.

Qualitative Results

Below we provide qualitative results for some tasks.

Visual Grounding

Image Captioning

Open-Ended VQA

<br></br>

Citation

If you find the work helpful, you can cite it using the following citation:

@article{
shukor2023unival,
title={Un{IVAL}: Unified Model for Image, Video, Audio and Language Tasks},
author={Mustafa Shukor and Corentin Dancette and Alexandre Rame and Matthieu Cord},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2023},
url={https://openreview.net/forum?id=4uflhObpcp},
note={}
}

<br></br>

Aknowledgment

This code is based mainly on the following repos:

• OFA
• Fairseq
• taming-transformers

We thank the authors for releasing their code.