Awesome

• This repository is only for study, please do not use for commercial.

Theano implementation of LSTM and CTC

• Support GPU accelaration. (Only when the model is complicated enough the GPU acceleration can be seen)
• Support LogCTC, which prevents from overflow issue
• Support batch training, which means that different width of images can be packed into a single mini-batch
• Support prediction alone script, useful to test and release.
• Currently not support CNN feature extraction, as there are plenty framework to do so. Raw pixel features in gray scale are feed directly into LSTM and CTC layers.

Data format

• x is a list of images. The size of each image should have identical height, and the width varies.
• x_mask is a matrix to indicate the valid region of each image. As in a mini-batch, images with different width are packed together, we need extra infomation to store the real width of each image. For example, [[1, 1, 1, 0, 0], [1, 1, 1, 1, 1]] represents 2 images, the width of first image is 3 and the second one is 5
• y is a matrix, each row is the corresponding label (sequence of chars) of each image.
• y_clip is a vector, similar as x_mask, however in more compact format. Each element represents the real length of corresponding label.

Installation

Ubuntu:

sudo apt-get update
sudo apt-get install -y libmagickwand-dev python-opencv
pip install -r requirements.txt

Mac OS X:

brew tap homebrew/science
brew install opencv
pip install -r requirements.txt

Add package path to your environment, for example export PYTHONPATH=/path/to/cnn-lstm-ctc/:$PYTHONPATH

OpenCV3 is needed

Get Data

• Pull all codes and data, including english_sentence.tar.gz (~21MB) in dataset folder
• Extract the files and run get_data.sh

cd /path/to/cnn-lstm-ctc/
cd dataset
tar -zxvf english_sentence.tar.gz
cd ..
sh get_data.sh

Train

cd /path/to/cnn-lstm-ctc/
sh train.sh

A typical validation accuracy curve within 200 epochs, lr=0.01, gamma=0.1, multistep_epoch=[100, 150, 175], cost 28.255 hours. (In this repository only 0.01% data are released due to privacy)

The best result occurs in epoch 188 with 67.152%

Predict

• After training converged and finished, pick the favorite model in snapshot folder, say, 188.pkl.
• Open predict.sh and modify model_path=snapshot/188.pkl

cd /path/to/cnn-lstm-ctc/
vim predict.sh
sh predict.sh

Others

• The first time to compile the train and val function in train/train.py takes long, maybe 10-15 mins
• For simple english word recognition task, with whole dataset, 92.8% accuracy can be obtained after 200 epoch.