Home

Awesome

Athena

Athena is an open-source implementation of end-to-end speech processing engine. Our vision is to empower both industrial application and academic research on end-to-end models for speech processing. To make speech processing available to everyone, we're also releasing example implementation and recipe on some opensource dataset for various tasks (Automatic Speech Recognition, Speech Synthesis, Voice activity detection, Wake Word Spotting, etc).

All of our models are implemented in Tensorflow>=2.0.1. For ease of use, we provide Kaldi-free pythonic feature extractor with Athena_transform.

Key Features

Versions

What's new

Discussion & Communication

We have set up a WeChat group for discussion. Please scan the QR and then the administrator will invite you to the group, if you want to join it.

<img src="https://github.com/JianweiSun007/athena/blob/athena-v0.2/docs/image/img1.png" width="250px">

1) Table of Contents

2) Installation

Athena can be installed based on Tensorflow2.3 and Tensorflow2.8 successfully.

pip install tensorflow-gpu==2.3.0

pip install -r requirements.txt

python setup.py bdist_wheel sdist

python -m pip install --ignore-installed dist/athena-2.0*.whl

pip install tensorflow-gpu==2.8.0

pip install -r requirements.txt

python setup.tf2.8.py bdist_wheel sdist

python -m pip install --ignore-installed dist/athena-2.0*.whl

3) Results

3.1) ASR

The performances of a part of models are shown as follow:

<details><summary>expand</summary><div>
ModelLMHKUSTAISHELL1 DatasetLibriSpeech DatasetGigaMISPModel link
CER%CER%WER%WER%CER%
devdevtestdev _cleandev _othertest_ cleantest_ otherdevtest-
transformerw21.64-5.13-----11.70-
w/o21.87-5.223.84-3.969.70---
transformer-u2w----------
w/o--6.38-------
conformerw21.33-4.95------50.50
w/o21.59-5.04-------
conformer-u2w----------
w/o--6.29-------
conformer-CTCw----------
w/o--6.60-------
</div></details>

To compare with other published results, see wer_are_we.md.

More details of U2, see ASR readme

3.2) TTS

Currently supported TTS tasks are LJSpeech and Chinese Standard Mandarin Speech Copus(data_baker). Supported models are shown in the table below: (Note:HiFiGAN is trained based on TensorflowTTS)

The performance of Athena-TTS are shown as follow:

<details><summary>expand</summary><div>
Traing DataAcoustic ModelVocoderAudio Demo
data_bakerTacotron2GLaudio_demo
data_bakerTransformer_ttsGLaudio_demo
data_bakerFastspeechGLaudio_demo
data_bakerFastspeech2GLaudio_demo
data_bakerFastspeech2HiFiGANaudio_demo
ljspeechTacotron2GLaudio_demo
</div></details>

More details see TTS readme

3.3) VAD

<details><summary>expand</summary><div>
TaskModel NameTraining DataInput SegmentFrame Error Rate
VADDNNGoogle Speech Commands Dataset V20.21s8.49%
VADMarbleNetGoogle Speech Commands Dataset V20.63s2.50%
</div></details>

More details see VAD readme

3.4) KWS

The performances on MISP2021 task1 dataset are shown as follow:

<details><summary>expand</summary><div>
KWS TypeModelModel DetailDataLossDevEval
StreamingCNN-DNN2 Conv+3 Dense60h pos+200h negCE0.314/
E2ECRNN2 Conv+2 biGRU60h pos+200h negCE0.209/
E2ECRNNConv+5 biLSTM60h pos+200h negCE0.186/
E2ECRNNConv+5 biLSTM170h pos+530h negCE0.178/
E2EA-TransformerConv+4 encoders+1 Dense170h pos+530h negCE&Focal0.1090.106
E2EA-ConformerConv+4 encoders+1 Dense170h pos+530h negCE&Focal0.1050.116
E2EAV-Transformer2 Conv+4 AV-encoders+1DenseA(170h pos+530h neg)+V(Far 124h)CE0.132/
</div></details>

More details you can see: KWS readme

3.5) CTC-Alignment

The CTC alignment result of one utterance is shown below, we can see the output of ctc alignment is with time delayed:

<details><summary>expand</summary><div> <div align="left"><img src="examples/align/ctc_alignment_demo.png" width="550"/></div> </div></details>

More details see: Alignment readme

3.6) Deploy

Athena-V2.0 deployment only support the ASR. All the experiments are conducted on a CPU machine with 2.10GHz and 104 logic cores. We evaluate the performance on AIShell datasets. The results are shown as follow:

<details><summary>expand</summary><div>
Logic CoreDecoder TypeBeamsizeRTFCharacter Accuracy
1BeamSearch10.088192.65%
100.253493.07%
200.453793.06%
1010.0479292.65%
100.113593.07%
200.174693.06%
1CTC Prefix BeamSearch10.054393.60%
100.0693.60%
200.090393.60%
1010.028393.60%
100.03893.60%
200.064193.60%
</div></details>

More detail see: Runtime readme

4)Run demo

We provide a quick experience method as follow:

cd athena
source tools/env.sh
#ASR test
# Batch decoding test
python athena/run_demo.py --inference_type asr --saved_model_dir examples/asr/aishell/models/freeze_prefix_beam-20220620 --wav_list test.lst
# One wav test
python athena/run_demo.py --inference_type asr --saved_model_dir examples/asr/aishell/models/freeze_prefix_beam-20220620 --wav_dir aishell/wav/test/S0764/BAC009S0764W0121.wav

#TTS test
python athena/run_demo.py --inference_type tts --text_csv examples/tts/data_baker/test/test.csv --saved_model_dir athena-model-zoo/tts/data_baker/saved_model

There are some pre-trained models and you can find at: Athena-model-zoo

More examples you can find at:

ASR examples

TTS examples

VAD examples

KWS examples

Alignment examples

C++ Decoder

Server

5) Supported Model architectures and reference

The Athena-v2.0 can support these architectures:

<details><summary>expand</summary><div>
Model NameTaskReferenced Papers
TransformerASRDong L, Xu S, Xu B. Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition[C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018: 5884-5888.
ConformerASRGulati A, Qin J, Chiu C C, et al. Conformer: Convolution-augmented transformer for speech recognition[J]. arXiv preprint arXiv:2005.08100, 2020.
Transformer-U2ASRYao Z, Wu D, Wang X, et al. Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit[J]. arXiv preprint arXiv:2102.01547, 2021.
Conformer-U2ASRYao Z, Wu D, Wang X, et al. Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit[J]. arXiv preprint arXiv:2102.01547, 2021.
AV_TransformerASR
AV_ConformerASR
FastspeechTTSRen Y, Ruan Y, Tan X, et al. Fastspeech: Fast, robust and controllable text to speech[J]. Advances in Neural Information Processing Systems, 2019, 32.
Fastspeech2TTSRen Y, Hu C, Tan X, et al. Fastspeech 2: Fast and high-quality end-to-end text to speech[J]. arXiv preprint arXiv:2006.04558, 2020.
Tacotron2TTSShen J, Pang R, Weiss R J, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions[C]//2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018: 4779-4783.
TTS_TransfprmerTTSLi N, Liu S, Liu Y, et al. Neural speech synthesis with transformer network[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2019, 33(01): 6706-6713.
MarblenetVADJia F, Majumdar S, Ginsburg B. Marblenet: Deep 1d time-channel separable convolutional neural network for voice activity detection[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021: 6818-6822.
DNNVADTashev I, Mirsamadi S. DNN-based causal voice activity detector[C]//Information Theory and Applications Workshop. 2016.
CNN-DNN, CRNN, A-Transformer, A-Conformer, AV-TransformerKWSXu Y, Sun J, Han Y, et al. Audio-Visual Wake Word Spotting System for MISP Challenge 2021[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 9246-9250.
</div></details>

6) Directory Structure

Below is the basic directory structure for Athena

<details><summary>expand</summary><div> 
|-- Athena
|   |-- data  # - root directory for input-related operations
|   |   |-- datasets  # custom datasets for ASR, TTS and pre-training
|   |-- layers  # some layers
|   |-- models  # some models
|   |-- tools # contains various tools, e.g. decoding tools
|   |-- transform # custom featureizer based on C++
|   |   |-- feats
|   |   |   |-- ops # c++ code on tensorflow ops
|   |-- utils # utils, e.g. checkpoit, learning_rate, metric, etc
|-- docker
|-- docs  # docs
|-- examples  # example scripts for ASR, TTS, etc
|   |-- asr  # each subdirectory contains a data preparation scripts and a run script for the task
|   |   |-- aishell
|   |   |-- hkust
|   |   |-- librispeech
|   |   |-- gigaspeech
|   |   |-- misp
|   |-- kws ## Word wake spotting
|   |   |-- misp
|   |   |-- xtxt
|   |   |-- yesno
|   |-- tts ## TTS examples
|   |   |-- data_baker
|   |   |-- ljspeech
|   |-- vad #VAD example
|       |--google_dataset_v2
|-- tools  # need to source env.sh before training
</div></details>

#7) Acknowledgement

We want to thank Espnet, Wenet, TensorFlowTTS, NeMo, etc. These great projects give us lots of references and inspirations!