Awesome

Code for Self-Critical Reasoning for Robust Visual Question Answering (NeurIPS 2019 Spotlight)

This repo contains codes for ''Self-Critical Reasoning for Robust Visual Question Answering'' with VQA-X human textual explanations This repo contains code modified from here, many thanks!

Prerequisites

Python 3.7.1 PyTorch 1.1.0 spaCy (we use en_core_web_lg spaCy model) h5py, pickle, json, cv2

Preprocessing

Please download the detection features from this google drive and put it to 'data' folder Please run bash tools/download.sh to download other useful data files including VQA QA pairs and Glove embeddings Please run bash tools/preprocess.sh to preprocess the data mkdir saved_models

Training

The training propocess is split to three stage or two stage:

Three stage version (pretrain on CP, fine-tune using influential strengthening loss and fine-tune with both.)

(1) Pretrain on VQA-CP train dataset by runnning CUDA_VISIBLE_DEVICES=0 python main.py --load_hint -1 --use_all 1 --learning_rate 0.001 --split v2cp_train --split_test v2cp_test --max_epochs 40 After the pretraining you will have a saved model in saved_models named by the start training time. Alternatively, you can directly download a model from here.

(2) Pretrain using the influential strengthening loss Here, please replace the 86-th line in the train.py with your VQA-CP pretrained models. Then, please run the following line to strengthen the most influential object. CUDA_VISIBLE_DEVICES=0 python main.py --load_hint 0 --use_all 0 --learning_rate 0.00001 --split v2cp_train_vqx --split_test v2cp_test --max_epochs 12 --hint_loss_weight 20 After the pretraining you will have anthor saved model in saved_models named by the start training time. Alternatively, you can directly download a model from here.

(3) Training with the self-critical objectives. Here, please replace the 82-th line in the train.py with your influence strengthened pretrained models. Then, please run the following line for training. CUDA_VISIBLE_DEVICES=0 python main.py --load_hint 1 --use_all 0 --learning_rate 0.00001 --split v2cp_train_vqx --split_test v2cp_test --max_epochs 5 --hint_loss_weight 20 --compare_loss_weight 1500 Alternatively, you can directly download a model from here.