Awesome
CoDraw Models
This repository contains models for the Collaborative Drawing (CoDraw) task.
Installation
Dependencies:
- Python 3.6 or later
- PyTorch 0.4.0
- IMPORTANT: Our pre-trained models are not compatible with PyTorch 0.4.1. Please use version 0.4.0 exactly.
Pillow
5.1.0 (or compatible)editdistance
0.4 (or compatible)
You will need to clone both this repository and the CoDraw dataset repository into side-by-side folders. The README for the dataset repository contains additional instructions for downloading required dataset files.
The following relative paths should be reachable from within this repository:
$ ls -d ../CoDraw/Pngs
../CoDraw/Pngs
$ ls -d ../CoDraw/dataset/CoDraw_1_0.json
../CoDraw/dataset/CoDraw_1_0.json
Pre-trained models
Pre-trained models can be downloaded from this link: models.tar.gz. The archive contains a models/
folder that should be placed at the root of this repository.
Usage
Automated evaluation
After downloading our pre-trained models, run python eval_automatic.py
to calculate the machine-machine and script-based scene similarity numbers that we report in our paper.
Training new models
Before training any models, please make sure that the models
and rl_models
folders exist within this repo: mkdir -p models rl_models
.
Each of the following commands trains a subset of the models we report in the paper:
python baseline1_train.py
python baseline2_train.py
python baseline3_train.py
python baseline4_train.py
Trained models are loaded by the function load_baseline1
in baseline1_models.py
and its counterparts in baseline2_models.py
, baseline3_models.py
, baseline4_models.py
. Note that all of these functions use hard-coded paths that match our pre-trained model release; you will probably need to change these paths if you train your own models. Also note that the training process of some of the later models relies on the existence of earlier ones.
Evaluating playing with humans
The transcripts for the human-machine evaluation reported in our paper are in the transcripts-eval-v1.json
file in this repository. To compute the scene similarity scores we report in our paper, update the TRANSCRIPTS_PATH
variable in eval_transcripts.py
and then run python eval_transcripts.py
.
Reference
If you find this code useful in your research, we'd really appreciate it if you could cite the following paper:
@article{CoDraw,
author = {Kim, Jin-Hwa and Kitaev, Nikita and Chen, Xinlei and Rohrbach, Marcus and Tian, Yuandong and Batra, Dhruv and Parikh, Devi},
journal = {arXiv preprint arXiv:1712.05558},
title = {{CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication}},
url = {http://arxiv.org/abs/1712.05558},
year = {2019}
}
License
This repository is licensed under Creative Commons Attribution-NonCommercial 4.0 International Public License, as found in the LICENSE file.