Awesome

A Structured Self-attentive Sentence Embedding

Tensorflow Implementation of "A Structured Self-attentive Sentence Embedding" (ICLR 2017).

Usage

Data

• AG's news topic classification dataset.
• The csv files (in my data directory) were available from here.

Train

• "GoogleNews-vectors-negative300" is used as pre-trained word2vec model.

• Display help message:

  $ python train.py --help
  

  train.py:
  	--[no]allow_soft_placement: Allow device soft device placement
  		(default: 'true')
  	--batch_size: Batch Size
  		(default: '64')
  		(an integer)
  	--checkpoint_every: Save model after this many steps
  		(default: '100')
  		(an integer)
  	--d_a_size: Size of W_s1 embedding
  		(default: '350')
  		(an integer)
  	--dev_sample_percentage: Percentage of the training data to use for validation
  		(default: '0.1')
  		(a number)
  	--display_every: Number of iterations to display training info.
  		(default: '10')
  		(an integer)
  	--embedding_dim: Dimensionality of word embedding
  		(default: '300')
  		(an integer)
  	--evaluate_every: Evaluate model on dev set after this many steps
  		(default: '100')
  		(an integer)
  	--fc_size: Size of fully connected layer
  		(default: '2000')
  		(an integer)
  	--hidden_size: Size of LSTM hidden layer
  		(default: '256')
  		(an integer)
  	--learning_rate: Which learning rate to start with.
  		(default: '0.001')
  		(a number)
  	--[no]log_device_placement: Log placement of ops on devices
  		(default: 'false')
  	--max_sentence_length: Max sentence length in train/test data
  		(default: '50')
  		(an integer)
  	--num_checkpoints: Number of checkpoints to store
  		(default: '5')
  		(an integer)
  	--num_epochs: Number of training epochs
  		(default: '10')
  		(an integer)
  	--p_coef: Coefficient for penalty
  		(default: '1.0')
  		(a number)
  	--r_size: Size of W_s2 embedding
  		(default: '30')
  		(an integer)
  	--train_dir: Path of train data
  		(default: 'data/train.csv')
  	--word2vec: Word2vec file with pre-trained embeddings
  

• Train Example (with word2vec):

  $ python train.py --word2vec "GoogleNews-vectors-negative300.bin"
  

Evalutation

• You must give "checkpoint_dir" argument, path of checkpoint(trained neural model) file, like below example.

• If you don't want to visualize the attention, give option like --visualize False.

• Evaluation Example:

  $ python eval.py --checkpoint_dir "runs/1523902663/checkpoints/"
  

Results

1) Accuracy test data = 0.920789

2) Visualization of Self Attention

Reference

• A Structured Self-attentive Sentence Embedding (ICLR 2017), Z Lin et al. [paper]
• flrngel's Self-Attentive-tensorflow github repository