Awesome

BigCiDian

1. Goal

This project is an attempt to create a pronunciation lexicon covering both English and Chinese words in a unified phoneset for ASR applications.

P.S. "CiDian" means "lexicon" in Chinese.

typical use cases in Chinese ASR applications:

你手机上都装了什么 APP ?
APPLE 的新 MACBOOK PRO 真漂亮
上个月 PRADA 出了款新包包
手机开了 GPRS 导航
世界杯 H 组小组赛

2. Phoneset

The unified phoneset should be a simple and precise phoneset that covers both languages. Note that the mapping listed below are heavily based on IPA.

2.1 English Phoneset Mapping

English entries are derived from CMUDict 0.7b, hence we need a mapping from ARPA phoneset to target phoneset.

notes: If you find anything that doesn't make sense in the mapping table, please let me know, thanks

2.2 Chinese PinYin Mapping

Chinese entries are extracted from DaCiDian project

Here is a PinYin to IPA mapping from educational prospective: https://resources.allsetlearning.com/chinese/pronunciation/Pinyin_chart

With a few mapping modifications and symbolic adaptations, here is the final PinYin to target phoneset mapping

2.3 tone

There are normally 5 tones in Chinese PinYin system ranging from 0 ~ 4. However there is no tone definition in English. In BigCiDian, Chinese tonal information is retained and merged with untoned English, so the resulting phoneset may contain 6 tonal variation(1 from English and 5 from Chinese):

e.g. for phoneme *ai*

1. HI -> h ai
2. 哎 -> ai_0
3. 掰 -> b ai_1
4. 还 -> h ai_2
5. 凯 -> k ai_3
6. 外 -> w ai_4

2.4 the unified phoneset

The final unified bi-lingual phoneset details are listed below:

So overall there are 56 phonemes in the unified phoneset(regardless of tones).

Theoretically some phonemes can be split with smaller granularity(eg. au->a u, ɔi->ɔ i, an->a n ...), hence making the phoneset even more compact. But it is a common practice that larger acoustic modeling units are beneficial for Chinese ASR accuracy, and the existence of decision-tree based state-tying, makes base phoneset size less irrelevant to ASR problem.

I may or may not change the unified phoneset in the future, currently it seems to be sufficient for my purpose.

3. Usage

sh run.sh should give you a ready-to-use bi-lingual ASR lexicon (lexicon.txt), and a phoneset list(phones.list) in project root directory.

4. Extend entries

To extend the final lexicon with entries of your own interest(say "IPHONE", "华为P30"), you can either:

• add those entries into the very bottom sources(CMUDict and DaCiDian)

or:

• maintain a seperate extension-lexicon, and merge it with main lexicon automatically generated above.

5. Experiment result

In AISHELL-2 Mandarin ASR task, replacing Chinese lexicon(DaCiDian) with multilingual CN-EN lexicon(BigCiDian), details are showed below:

For DaCiDian, system performance:

----- test -----:
%WER 44.39 [ 21986 / 49532, 338 ins, 2085 del, 19563 sub ] exp/mono/decode_test/cer_9_0.0
%WER 24.25 [ 12011 / 49532, 393 ins, 792 del, 10826 sub ] exp/tri1/decode_test/cer_12_0.0
%WER 22.13 [ 10963 / 49532, 396 ins, 644 del, 9923 sub ] exp/tri2/decode_test/cer_12_0.0
%WER 19.29 [ 9555 / 49532, 263 ins, 640 del, 8652 sub ] exp/tri3/decode_test/cer_13_0.5
%WER 8.33 [ 4125 / 49532, 84 ins, 192 del, 3849 sub ] exp/chain/tdnn_1a/decode_test/cer_8_0.5

For BigCiDian, system performance:

%WER 43.92 [ 21754 / 49532, 405 ins, 1574 del, 19775 sub ] exp/mono/decode_test/cer_7_0.0
%WER 22.54 [ 11163 / 49532, 406 ins, 652 del, 10105 sub ] exp/tri1/decode_test/cer_11_0.0
%WER 21.09 [ 10445 / 49532, 377 ins, 609 del, 9459 sub ] exp/tri2/decode_test/cer_12_0.0
%WER 18.47 [ 9148 / 49532, 265 ins, 621 del, 8262 sub ] exp/tri3/decode_test/cer_13_0.5
%WER 8.22 [ 4072 / 49532, 68 ins, 260 del, 3744 sub ] exp/chain/tdnn_1a/decode_test/cer_9_0.5

Conclusion

• It shows that BigCiDian only gives slightly better results than DaCiDian.
• But more importantly, BigCiDian turns a pure Chinese ASR system to multiligual system, which is pretty much the case in nowadays Chinese ASR applications.

THE END