Home

Awesome

VisDial AMT Chat

Source for the two-person chat interface used to collect the VisDial dataset (arxiv.org/abs/1611.08669) on Amazon Mechanical Turk.

<!-- A demo is available [here][12] (open in two separate tabs to be paired for conversation). -->

VisDial AMT Interface

If you find this code useful, consider citing our work:

@inproceedings{visdial,
  title={{V}isual {D}ialog},
  author={Abhishek Das and Satwik Kottur and Khushi Gupta and Avi Singh
    and Deshraj Yadav and Jos\'e M.F. Moura and Devi Parikh and Dhruv Batra},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2017}
}

How things work

The real-time chat interface is built using Node.js and Socket.io. We use Redis to maintain a pool of images for live HITs and all data finally resides in a MySQL database.

A database table stores images from the COCO dataset each with a randomly picked caption. A batch of images from this table are then pushed to a Redis list for launching HITs. The web server corresponding to the chat interface pairs two AMT workers, assigns them roles (questioner or answerer) and shows corresponding interface, picks an image from the Redis list to collect data on and saves their conversation in the database, also marking that image as 'complete' once the HIT is done. This happens in parallel so workers aren't left waiting, and the server ensures workers have unique ids. Disconnects are handled gracefully — remaining worker is asked to continue asking questions or providing facts (captions) up to 10 messages. Once the HITs are complete, scripts in mturk_scripts/approve can be used to review, approve, reject HITs and pay workers.

Node.js

Pops images from the Redis list, retrieves relevant data from the MySQL tables, renders the chat interface and pairs workers on AMT, saves submitted data to MySQL tables

MTurk Scripts

Scripts to set up MySQL database, populate Redis list, and approve/reject HITs

Approving/Rejecting HITs (mturk_scripts/approve)

Contributors

Acknowledgements

Parts of this code (MTurk scripts) are adapted from @jcjohnson's simple-amt project.

License

BSD