Awesome

datasets-CMU_Wilderness

CMU Wilderness Multilingual Speech Dataset

A dataset of over 700 different languages providing audio, aligned text and word pronunciations. On average each language provides around 20 hours of sentence-lengthed transcriptions. Data is mined from read New Testaments from http://www.bible.is/

List of Languages with relative scores of accuracy of alignment

http://festvox.org/cmu_wilderness/

Map of Languages geopositioned

http://festvox.org/cmu_wilderness/map.html

Language List

The file LangList.txt has a list of all processed languages with features as space separated fields

1  LANGID six letter language id from bible.is
2  TLC    three letter language code (iso 639-3)
3  WIKI   Wikipedia link to language description
4  START  start url at bible.is
5  LAT    geolocated latitude
6  LONG   geolocated longitude
7  #utt0  Number utterances found in Pass 0 (cross-lingual alignment)
8  MCD0   Mel Cepstral Distortion score for Pass 0 (smaller is better)
9  #utt1  Number utterances found in Pass 1 (in-language alignment)
10 MCD1   Mel Cepstral Distortion score for Pass 1 (smaller is better)
11 Dur    HH:MM:SS duration of alignment data (from Pass 1)
12 MCDB   Mel Cepstral Distortion score for base CG synthesizer
13 MCDR   Mel Cepstral Distortion score for Random Forest CG synthesizer
14+ NAME  Text name of language (may be multiple fields)

Prerequisites

Ubuntu (and related) prerequisites:

sudo apt-get install git build-essential libncurses5-dev sox wget
sudo apt-get install csh ffmpeg html2text

Note that the ffmpeg package is sometimes called avconv (you need to update bin/do_found accordingly if you only have avconv and not ffmpeg).

Clone the repository

git clone https://github.com/festvox/datasets-CMU_Wilderness
cd datasets-CMU_Wilderness

Make Dependencies

Builds the FestVox voice building tools in build/ and sets up the environment variable settings in festvox_env_settings

./bin/do_found make_dependencies

Create Alignments For A Language

Because we cannot redistribute the audio from bible.is, you must download that data directly, then build the alignments using the indices we distribute.

Alignments (short waveforms plus transcripts) may be recreated for a language from the packed versions in the indices/ directory. You need to know the six letter code for the language (see LangList for mappings). In this example we use NANTTV (Hokkien) to illustrate the commands, but you should substitute the code for your desired language.

nohup ./bin/do_found fast_make_align indices/NANTTV.tar.gz &

This will unpack the indices in the NANTTV directory, download the data from bible.is (unless it is already in downloads/NANTTV/download/) then reconstruct the aligned data in NANTTV/aligned/wav/ and NANTTV/aligned/etc/ This process will take around 30 minutes depending on your internet connection, and the speed of your machine.

Create Text To Speech Model

Given the alignments in aligned/ you can build a speech synthesizer for Festival (and Flite) as follows.

cd NANTTV
nohup ../bin/do_found make_tts &
../bin/do_found get_voices

Will build a Random Forest Clustergen synthesis model for Festival and Flite in NANTTV/voices/ This will take at least 48 hours on a 12 core machine.

Create Speech To Text Model

You can use the waveforms in NANTTV/aligned/wav/ and transcriptions in NANTTV/aligned/etc/trascription.txt. The file NANTTV/aligned/etc/txt.done.data also has an alignment score (lower is better) for utterance. If you want a pronunciation lexicon and transcription without punctuation you execute

cd NANTTV
nohup ../bin/do_found make_asr &

This does not (yet) build a model, but gives a punctuation free transcription file in NANTTV/aligned/etc/transcription_nopunct.txt and a pronunciation lexicon in NANTTV/aligned/etc/pronunciation_lex

Creating New Alignments

You can do the full alignment creation if you want. Our alignments certainly can be improved on with better acoustic models, pronunciations etc. If you are interested in re-aligning you can do so with the command

nohup ./bin/do_found full_make_align http://listen.bible.is/NANTTV/Matt/1/D &

This may take around 7 days on a 12 core machine. It needs about 150GB of diskspace (which can be reduced with the command ../bin/do_found tidy_up at the end to about 20GB). The alignments themselves are usually around 2GB.

Aligning all 700 languages will take around 13 years on a single machine.

Creating phone level alignments for all utterances

Given the sentence level alignments generated by fast_make_align or full_make_align, you can generate phone level alignments for every utterance with

cd NANTTV
nohup ../bin/do_found make_phone_alignments &

This may take several hours to run the hmm aligner. The generated phone alignments are in v_ph_aligns/lab/*.lab in Xlabel ascii format. Full Festival utterances with all alignments and linked with syllables, words etc are available in v_ph_aligns/festival/utts/*.utt. See http://www.festvox.org/bsv/x1902.html for an example of dumping features and model building.

Citations

For more details, see

Alan W Black "CMU Wilderness Multilingual Speech Dataset" ICASSP 2019, Brighton, UK.

Acknowledgments

This dataset was prepared by Alan W Black (awb@cs.cmu.edu) with substantial help from a large number of CMU students. We also would like to thank various members of the CMU community, especially Florian Metze, for access to CPU resources to help calculate the alignments. This work was in part funded by the DARPA Lorelei Program.