Awesome

OFA is a unified sequence-to-sequence pretrained model (support English and Chinese) that unifies modalities (i.e., cross-modality, vision, language) and tasks (finetuning and prompt tuning are supported): image captioning (1st at the MSCOCO Leaderboard), VQA (link), visual grounding, text-to-image generation, text classification, text generation, image classification, etc. We provide step-by-step instructions for pretraining and finetuning and corresponding checkpoints (check official ckpt [EN|CN] or Hugging Face ckpt).

We sincerely welcome contributions to our project. Feel free to contact us or send us issues / PRs! <br></br>

Online Demos

We provide online demo via Hugging Face Spaces for you to interact with our pretrained and finetuned models. Below are the links to the demos:

• Image Captioning [ModelScope | Spaces]
• Visual Grounding [ModelScope | Spaces]
• Visual Question Answering [ModelScope | Spaces]
• Text-to-Image Generation [ModelScope | Spaces]
• Generic Interface [Spaces]
• Chinese OCR [ModelScope | Spaces]

Also we provide Colab notebooks for you to better perceive the procedures. Click here to check them out! <br></br>

Use in Hugging Face Transformers

We support the inference of OFA in Hugging Face Transformers. Check the README and Colab Notebook for more information. Codes are released in this branch https://github.com/OFA-Sys/OFA/tree/feature/add_transformers <br><br>

News

• 2023.5.11: Two papers (OFA-OCR and OFA-prompt) are accepted by ACL. The evaluation scripts and checkpoints of OFA-OCR are released.
• 2023.1.11: Released MuE (https://arxiv.org/abs/2211.11152), which significantly accelerates OFA with little performance degradation. Many thanks to the first author, Shengkun Tang (@Tangshengku). See the branch feature/MuE and PR for more information.
• 2022.12.20: Released OFA-OCR, a model for Chinese text recognition based on OFA. Check our paper and demo.
• 2022.12.7: Released the MMSpeech an ASR pre-training method based on OFA. Check our paper here! Please see the README_mmspeech.md for further details.
• 2022.8.16: Released the Chinese version of OFA. OFA-CN needs only switching to bpe_dir=../../utils/BERT_CN_dict and bpe=bert and using our provided Chinese checkpoints in checkpoints_cn.md. Temporarily, we only provide base-size and large-size pretrained checkpoints and finetuned checkpoints on MUGE Caption and the Chinese version of RefCOCO(-/+/g) (to release soon).
• 2022.8.5: Released support of prompt tuning for OFA. Check our paper here! Please see the prompt_tuning.md for further details.
• 2022.7.7: Updated support of OFA on Hugging Face transformers (fixed bugs in forward, add sequence generator from Fairseq to ensure performance, etc.). Refer to the doc transformers.md and the branch feature/add_transformers.
• 2022.6.17: Released the pretrained checkpoint of OFA-Huge. To use it, set --arch=ofa_huge in the script.
• 2022.5.15: OFA was accepted by ICML 2022

Model Card

We list the parameters and pretrained checkpoints of OFAs below. For finetuned checkpoints, please refer to checkpoints.md.

Results

Below we demonstrate the results of OFAs on cross-modal understanding and generation.

Requirements

• python 3.7.4
• pytorch 1.8.1
• torchvision 0.9.1
• JAVA 1.8 (for COCO evaluation) <br></br>

Installation

git clone https://github.com/OFA-Sys/OFA
pip install -r requirements.txt

<br></br>

Datasets and Checkpoints

See datasets.md and checkpoints.md. <br></br>

Training & Inference

Below we provide methods for training and inference on different tasks. We provide both pretrained OFA-Large and OFA-Base in checkpoints.md. The scripts mentioned in this section are prepared for OFA-Large. For reproducing the downstreaming results of OFA-Base, we have also provided the corresponding finetuning and inference scripts for OFA-Base in the run_scripts/ folder.

We recommend that your workspace directory should be organized like this:

OFA/
├── checkpoints/
│   ├── ofa_base.pt
│   ├── ofa_large.pt
│   ├── caption_large_best_clean.pt
│   └── ...
├── criterions/
├── data/
├── dataset/
│   ├── caption_data/
│   ├── gigaword_data/
│   └── ...
├── fairseq/
├── models/
├── run_scripts/
├── tasks/
├── train.py
├── trainer.py
└── utils/

Image Processing

To ensure the efficiency of processing data, we did not store images with small files, but instead we encode them to base64 strings. Transforming image files to base64 strings is simple. Run the following code:

from PIL import Image
from io import BytesIO
import base64

img = Image.open(file_name) # path to file
img_buffer = BytesIO()
img.save(img_buffer, format=img.format)
byte_data = img_buffer.getvalue()
base64_str = base64.b64encode(byte_data) # bytes
base64_str = base64_str.decode("utf-8") # str

Pretraining

Below we provide methods for pretraining OFA.

Image Captioning

We provide procedures to reproduce our results of image captioning on our paper below.

Text-to-Image Generation

This part provides procedures for the finetuning and inference of text-to-image generation. See below.

Visual Question Answering

Here we provide the finetuning and inference codes to reproduce the VQAv2 result reported in our paper (test-std 80.02). We believe much improvement on accuracy can still be achieved based on this codebase :)

Visual Grounding (Referring Expression Comprehension)

Here provides procedures for you to prepare data, train, and evaluate your model on visual grounding.

Visual Entailment

We provide steps for you to reproduce our results in visual entailment. See the details below.

GLUE

Here we provide steps for you to finetune and evaluate our model on language understanding tasks. We demonstrate our practice for the GLUE benchmark.

Image Classification on ImageNet-1K

We provide the finetuning and inference codes which reproduce 85.0 ImageNet-1K accuracy, slightly better than reported in our paper.

Gigaword

We provide steps for you to reproduce our results in Gigaword. See the details below.

<br></br>

Gallery

Below we provide examples of OFA in text-to-image generation and open-ended VQA. Also, we demonstrate its performance in unseen task (Grounded QA) as well as unseen domain (Visual Grounding on images from unseen domains).

Text-to-Image Generation

Open-Ended VQA

Grounded QA (unseen task)

Visual Grounding (unseen domain)

<br></br>

Related Codebase

• Fairseq
• taming-transformers <br></br>

Getting Involved

Feel free to submit Github issues or pull requests. Welcome to contribute to our project!

To contact us, never hestitate to send an email to zheluo.wp@alibaba-inc.com or junyang.ljy@alibaba-inc.com! <br></br>

Citation

Please cite our papers if you find them helpful :)

@article{wang2022ofa,
  author    = {Peng Wang and
               An Yang and
               Rui Men and
               Junyang Lin and
               Shuai Bai and
               Zhikang Li and
               Jianxin Ma and
               Chang Zhou and
               Jingren Zhou and
               Hongxia Yang},
  title     = {OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence
               Learning Framework},
  journal   = {CoRR},
  volume    = {abs/2202.03052},
  year      = {2022}
}

<br></br>

@article{ofa_ocr,
  author       = {Junyang Lin and
                  Xuancheng Ren and
                  Yichang Zhang and
                  Gao Liu and
                  Peng Wang and
                  An Yang and
                  Chang Zhou},
  title        = {Transferring General Multimodal Pretrained Models to Text Recognition},
  journal      = {CoRR},
  volume       = {abs/2212.09297},
  year         = {2022}
}

<br><br>

@article{ofa_prompt,
  author       = {Hao Yang and
                  Junyang Lin and
                  An Yang and
                  Peng Wang and
                  Chang Zhou and
                  Hongxia Yang},
  title        = {Prompt Tuning for Generative Multimodal Pretrained Models},
  journal      = {CoRR},
  volume       = {abs/2208.02532},
  year         = {2022}
}

<br><br>

@article{mmspeech,
  title={MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognition},
  author={Zhou, Xiaohuan and Wang, Jiaming and Cui, Zeyu and Zhang, Shiliang and Yan, Zhijie and Zhou, Jingren and Zhou, Chang},
  journal={arXiv preprint arXiv:2212.00500},
  year={2022}
}