Home

Awesome

End-to-end Automatic Speech Recognition Systems - PyTorch Implementation

This is an open source project (formerly named Listen, Attend and Spell - PyTorch Implementation) for end-to-end ASR by Tzu-Wei Sung and me. Implementation was mostly done with Pytorch, the well known deep learning toolkit.

The end-to-end ASR was based on Listen, Attend and Spell<sup>1</sup>. Multiple techniques proposed recently were also implemented, serving as additional plug-ins for better performance. For the list of techniques implemented, please refer to the highlights, configuration and references.

Feel free to use/modify them, any bug report or improvement suggestion will be appreciated. If you find this project helpful for your research, please do consider to cite our paper, thanks!

Highlights

<p align="center"> <img src="tests/sample_data/demo.png" width="570" height="300"> </p>

You may checkout some example log files with TensorBoard by downloading them from coming soon

Dependencies

Instructions

Step 0. Preprocessing - Generate Text Encoder

You may use the text encoders provided at tests/sample_data/ and skip this step.

The subword model is trained with sentencepiece. As for character/word model, you have to generate the vocabulary file containing the vocabulary line by line. You may also use util/generate_vocab_file.py so that you only have to prepare a text file, which contains all texts you want to use for generating the vocabulary file or subword model. Please update data.text.* field in the config file if you want to change the mode or vocabulary file. For subword model, use the one ended with .model as vocab_file.

python3 util/generate_vocab_file.py --input_file TEXT_FILE \
                                    --output_file OUTPUT_FILE \
                                    --vocab_size VOCAB_SIZE \
                                    --mode MODE

For more details, please refer to python3 util/generate_vocab_file.py -h.

Step 1. Configuring - Model Design & Hyperparameter Setup

All the parameters related to training/decoding will be stored in a yaml file. Hyperparameter tuning and experiments can be managed easily this way. See documentation and examples for the exact format. Note that the example configs provided were not fine-tuned, you may want to write your own config for best performance.

Step 2. Training (End-to-end ASR or RNN-LM)

Once the config file is ready, run the following command to train end-to-end ASR (or language model)

python3 main.py --config <path of config file> 

For example, train an ASR on LibriSpeech and watch the log with

# Checkout options available
python3 main.py -h
# Start training with specific config
python3 main.py --config config/libri/asr_example.yaml
# Open TensorBoard to see log
tensorboard --logdir log/
# Train an external language model
python3 main.py --config config/libri/lm_example.yaml --lm

All settings will be parsed from the config file automatically to start training, the log file can be accessed through TensorBoard. Please notice that the error rate reported on the TensorBoard is biased (see issue #10), you should run the testing phase in order to get the true performance of model. Options available in this phase include the followings

OptionsDescription
configPath of config file.
seedRandom seed, note this is an option that affects the result
nameExperiments for logging and saving model. <p> By default it's <name of config file>_<random seed>
logdirPath to store training logs (log files for tensorboard), default log/.
ckpdirThe directory to store model, default ckpt/.
njobsNumber of workers used for data loader, consider increase this if you find data preprocessing takes most of your training time, default using 6.
no-pingDisable the pin-memory option of pytorch dataloader.
cpuCPU-only mode, not recommended, use it for debugging.
no-msgHide all message from stdout.
lmSwitch to rnnlm training mode.
testSwitch to decoding mode (do not use during training phase)
cudnn-ctcUse CuDNN as the backend of PyTorch CTC. Unstable, see this issue, not sure if solved in latest Pytorch with cudnn version > 7.6

Step 3. Speech Recognition & Performance Evaluation

To test a model, run the following command

python3 main.py --config <path of config file> --test --njobs <int>

Please notice that the decoding is performed without batch processing, use more workers to speedup at the cost of using more RAM. By default, recognition result will be stored at result/<name>/ as two csv files with auto-naming according to the decoding config file. output.csv will store the best hypothesis provided by ASR and beam.csv will recored the top hypotheses during beam search. The result file may be evaluated with eval.py. For example, test the example ASR trained on LibriSpeech and check performance with

python3 main.py --config config/libri/decode_example.yaml --test --njobs 8
# Check WER/CER
python3 eval.py --file result/asr_example_sd0_dev_output.csv

Most of the options work similar to training phase except the followings:

OptionsDescription
testMust be enabled
configPath to the decoding config file.
outdirPath to store decode result.
njobsNumber of threads used for decoding, very important in terms of efficiency. Large value equals fast decoding yet RAM/GPU RAM expensive.

Troubleshooting

ToDo

Acknowledgements

Reference

  1. Listen, Attend and Spell, W Chan et al.
  2. Neural Machine Translation of Rare Words with Subword Units, R Sennrich et al.
  3. Attention-Based Models for Speech Recognition, J Chorowski et al.
  4. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks, A Graves et al.
  5. Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning, S Kim et al.
  6. Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM, T Hori et al.

Citation

@inproceedings{liu2019adversarial,
  title={Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model},
  author={Liu, Alexander and Lee, Hung-yi and Lee, Lin-shan},
  booktitle={Acoustics, Speech and Signal Processing (ICASSP)},
  year={2019},
  organization={IEEE}
}

@misc{alex2019sequencetosequence,
    title={Sequence-to-sequence Automatic Speech Recognition with Word Embedding Regularization and Fused Decoding},
    author={Alexander H. Liu and Tzu-Wei Sung and Shun-Po Chuang and Hung-yi Lee and Lin-shan Lee},
    year={2019},
    eprint={1910.12740},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}