Awesome

VIBIKNet

This repository contains the code for the Visual Bidirectional Kernelized Network for Visual Question Answering, presented at the VQA Challenge at CVPR'16. With this module, you can replicate our experiments and easily deploy new models. VIBIKNet is built upon the Keras (version 1.2) framework and tested for the Theano backend.

Installation

VIBIKNet requires the following libraries:

• Our version of Keras v1.0.4 or newer
• Multimodal Keras Wrapper v0.1 or newer
• VQA challenge evaluation scripts

Additionally, if you want to run the tutorials and the visualization module, you'll need:

• Jupyter Notebook
• Matplotlib

If you want to extract KCNN features you will need (see the following README for more info):

• Matlab 2014a or newer or Octave
• Caffe. Download into ~/code/caffe folder or change path in extractCNNfeatures.m
• EdgeBoxes object detection. Download into /repository_root/edges folder.
• Piotr Dollar toolbox. Download into /repository_root/piotr_toolbox folder.
• Inria's Yael library for Matlab. Download into /repository_root/yael folder and compile for you system. See yael's release notes for more information.

If you want to use pretrained word embeddings, you can either train them by yourself using Glove or Word2Vec, or download pretrained word embeddings (recommended):

• Pretrained Glove vectors
• Pretrained Word2Vec vectors

How to use

• For extracting KCNN features, see the following README.
• For training a new model, follow the train README.
• For visualizing the results, follow the visualize_results notebook.

VIBIKNet model at the CVPR VQA Challenge

See CVPR poster here.

Examples

These answers have been automatically generated by VIBIKNet:

Project citation

If you use this project, please, cite the following publication:

Bolaños, M., Peris, Á., Casacuberta, F., & Radeva, P. 
VIBIKNet: Visual Bidirectional Kernelized Network for Visual Question Answering
Iberian Conference on Pattern Recognition and Image Analysis, IbPRIA '17 (In press).

References

Liu Z. 
Kernelized Deep Convolutional Neural Network for Describing Complex Images. 
arXiv preprint arXiv:1509.04581. 2015 Sep 15.

Peris Á, Bolaños M, Radeva P, Casacuberta F. 
Video Description using Bidirectional Recurrent Neural Networks. 
arXiv preprint arXiv:1604.03390. 2016 Apr 12.

Malinowski M, Rohrbach M, Fritz M. 
Ask your neurons: A neural-based approach to answering questions about images. 
In Proceedings of the IEEE International Conference on Computer Vision 2015 (pp. 1-9).

About

Joint collaboration between the Computer Vision at the University of Barcelona (CVUB) group at Universitat de Barcelona-CVC and the PRHLT Research Center at Universitat Politècnica de València.

Contact

Marc Bolaños (web page): marc.bolanos@ub.edu

Álvaro Peris: lvapeab@prhlt.upv.es