Awesome

Torch Seq2Seq Intent Parsing

Intent parsing framed as human to computer language translation.

The model uses the seq2seq encoder-decoder model with two decoders: one to decode the intent and slot placeholders (setState $light.name $light.state), and one to align input words with slot values ($light.name = { 0 0 0 1 1 0 }). Both are RNNs consisting of several GRU cells, and both use an attention mechanism over the encoder outputs.

Based on ideas from the following papers:

• Sequence to Sequence Learning with Neural Networks
• Neural Machine Translation by Jointly Learning to Align and Translate
• Incorporating Copying Mechanism in Sequence-to-Sequence Learning adds a "copy mode" in addition to the standard decoder (combined as a mixture) which uses an attention-like mechanism to decide which words to copy from the input
• Pointer Networks

Sampling example showing input, slot alignment, and output:

[sample]        may you send Nenita Service a text
$person =>      { 0 0 0 1 1 0 0 }
(sampled)  sendText ( $person = Nenita Service )

[sample]        what is the cost of pesos
$market =>      { 0 0 0 0 0 1 }
(sampled)  market getPrice ( $market = pesos )

[sample]        could you send Deloise Kamps Readme.md please
$person =>      { 0 0 0 1 1 0 0 }
$file   =>      { 0 0 0 0 0 1 0 }
(sampled)  sendFile ( $person = Deloise Kamps ) ( $file = Readme.md )

[sample]        turn off my lights thanks
$light.state    =>      { 0 1 0 0 0 }
$light.name     =>      { 0 0 1 1 0 }
(sampled)  lights setState ( $light.name = my lights ) ( $light.state = off )

Preparation

First download these 1.42 GB of 27 billion Twitter GloVe vectors from http://nlp.stanford.edu/data/wordvecs/glove.twitter.27B.zip and extract them to data/glove.twitter.27B.*.txt

Then run the cache-glove script to cache a subset of glove vectors (based on the sentence templates in templates.lua). This is to make it less painfully slow to start the training script (in case you want to quickly tweak a parameter and restart).

$ th cache-glove.lua

Training

$ th train.lua
Usage: [options]

-hidden_size         Hidden size of LSTM layer [200]
-glove_size          Glove embedding size [100]
-dropout             Dropout [0.1]
-learning_rate       Learning rate [0.0002]
-learning_rate_decay Learning rate decay [1e-06]
-max_length          Maximum output length [20]
-n_epochs            Number of epochs to train [10000]

With the small example training set you should get good results around 2500 epochs (about 5 minutes on a recent MacBook CPU).