Awesome
Interactive Attention Visualization
A small example of an interactive visualization for attention values as being used by transformer language models like GPT2 and BERT.
by Hendrik Strobelt and Sebastian Gehrmann for the SIDN IAP class at MIT, Jan 2020
<p align="center"> <img width="600" height="460" src="attnvis.gif"> </p>Preparation
- Install Anaconda or Miniconda
- run
conda env create -f environment.yml
to create a new environment calledattnvis
Running the demo
- activate conda environment:
conda activate attnvis
- run server:
python server.py
- visit http://localhost:8888/
Structure
api.py -- contains the interface to the pytorch/huggingface backend
server.py -- defines a REST interface for the api.py calls
client/* -- contains all client files
client/index.html -- main file including JS code
client/styles.css -- all CSS styles defined in here
client/tools.js -- helper functions
Libraries used
- Huggingface pytorch transformer for models
- Flask Python for REST interface
- D3js for javascript DOM manipulation
Thanks
Thanks to Ben Hoover and David Bau for testing the code. Supported by MIT-IBM Watson AI Lab.