Awesome
Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment
This repository contains the code and instructions to reproduce the results of the paper "Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment".
Getting Started
First, clone the repository and install the required dependencies:
git clone https://github.com/BrianG13/MismatchQuest.git
pip install -r requirements.txt
Textual-Visual Feedback Generation (Training data)
In order to reproduce our generation pipeline, we offer the following instructions:
Set your Google Cloud credentials
At consts.py
file you need to set the values for the constants:
GOOGLE_PROJECT_NAME
- The name of your project at Google Cloud
GOOGLE_APPLICATION_CREDENTIALS_PATH
- Absolute path to your credentials.json file. For more instructions how to get a .json
credentials file: link
Run the generation script
python run_congen_feedback.py --dataset <DATASET> --input_csv <INPUT_CSV_PATH> --output_dir <OUTPUT_DIR_PATH>
Arguments:
--dataset
- Dataset name to generate from:['coco', 'pickapic', 'imagereward', 'flickr30k', 'ade20k', 'open_images']
--input_csv
- Path for the input .csv file according to the dataset.--output_dir
- Output directory path to save the generated file.- [Optional]
--range
- You can split the .csv file by range using this argument as follows:LOW_RANGE,HIGH_RANGE
for example:--range 100,200
- [Optional]
--save_every_n
- Argument that marks saving frequency during generation.
Note: As explained in the paper, the image descriptions of ADE20K and OpenImages datasets (taken from LocalizedNarratives dataset) where pre-processed using another script in order to summarize the long description into a short caption.
Before generating Textual-Visual Feedback
for the mentioned datasets you need to run the following script:
python LocalizedNarrativesDescToCaption.py --input_jsonl_path <INPUT_JSONL_PATH> --output_dir <OUTPUT_DIR_PATH>
Arguments:
--input_jsonl_path
- Path for the input .jsonl downloaded from LocalizedNarratives--output_dir
- Output directory path to save the generated file.- [Optional]
--range
- You can split the .csv file by range using this argument as follows:LOW_RANGE,HIGH_RANGE
for example:--range 100,200
- [Optional]
--save_every_n
- Argument that marks saving frequency during generation.
SEETrue-Feedback dataset
SEETrue-Feedback a comprehensive alignment benchmark. It features 2,008 human-annotated instances that highlight textual and visual feedback. You can download the dataset at the following link