Awesome

COMICS

code to download comics data and train models described in The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives.

email miyyer@umd.edu and varunm@cs.umd.edu with any comments/problems/questions/suggestions.

dependencies:

requires python 2.7, lasagne, theano, h5py, cv2, glob2

to download / unzip / preprocess COMICS data:

bash setup.sh (downloads raw panel images, OCR transcriptions, etc., and preprocesses them into an hdf5 file)
if you don't want to download everything at once, you can download individual files at https://obj.umiacs.umd.edu/comics/index.html.

to train models after preprocessing (example for text cloze):

python models/text_cloze.py (make sure to run on GPU; see run.sh for our theano flags)
see description of hyperparameters by running python models/text_cloze.py --help
note that low-quality data is only filtered out in dev/test data (by throwing out examples with too many UNK tokens). during training, all data is used.

results:

method	text cloze easy	text cloze hard	visual cloze easy	visual cloze hard	character coherence
text only	63.4	52.9	55.9	48.4	68.2
image only	51.7	49.4	85.7	63.2	70.9
image text	68.6	61.0	81.3	59.1	69.3

if you use the COMICS data and/or code, please cite:

@InProceedings{Iyyer:Manjunatha-Comics2017,
    Title = {The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives},
    Booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
    Author = {Mohit Iyyer and Varun Manjunatha and Anupam Guha and Yogarshi Vyas and Jordan Boyd-Graber and Hal {Daum\'{e} III} and Larry Davis},
    Year = {2017},
}