Awesome

Feature

• VITS text-to-speech, voice conversion
• HuBert-soft VITS
• vits_chinese
• Bert-VITS2
• W2V2 VITS / emotional-vits dimensional emotion model
• GPT-SoVITS
• Support for loading multiple models
• Automatic language recognition and processing,set the scope of language type recognition according to model's cleaner,support for custom language type range
• Customize default parameters
• Long text batch processing
• GPU accelerated inference
• SSML (Speech Synthesis Markup Language) work in progress...

Online Demo

Thanks to Hugging Face!

Colab Notebook

Please note that different IDs may support different languages.speakers

• https://artrajz-vits-simple-api.hf.space/voice/vits?text=你好,こんにちは&id=164
• https://artrajz-vits-simple-api.hf.space/voice/vits?text=Difficult the first time, easy the second.&id=4
• excited:https://artrajz-vits-simple-api.hf.space/voice/w2v2-vits?text=こんにちは&id=3&emotion=111
• whispered:https://artrajz-vits-simple-api.hf.space/w2v2-vits?text=こんにちは&id=3&emotion=2077

https://user-images.githubusercontent.com/73542220/237995061-c1f25b4e-dd86-438a-9363-4bb1fe65b425.mov

Deployment

There are two deployment options to choose from. Regardless of the option you select, you'll need to import the model after deployment to use the application.

Docker Deployment (Recommended for Linux)

Step 1: Pull the Docker Image

Run the following command to pull the Docker image. Follow the prompts in the script to choose the necessary files to download and pull the image:

bash -c "$(wget -O- https://raw.githubusercontent.com/Artrajz/vits-simple-api/main/vits-simple-api-installer-latest.sh)"

The default paths for project configuration files and model folders are /usr/local/vits-simple-api/.

Step 2: Start

Run the following command to start the container:

docker-compose up -d

Image Update

To update the image, run the following commands:

docker-compose pull

Then, restart the container:

docker-compose up -d

Virtual Environment Deployment

Step 1: Clone the Project

Clone the project repository using the following command:

git clone https://github.com/Artrajz/vits-simple-api.git

Step 2: Install Python Dependencies

It is recommended to use a virtual environment with Python version 3.10 for this project. Run the following command to install the Python dependencies required for the project:

If you encounter issues installing certain dependencies, please refer to the common problems outlined below.

pip install -r requirements.txt

Step 3: Start

Run the following command to start the program:

python app.py

Windows Quick Deployment Package

Step 1: Download and Extract the Deployment Package

Go to the releases page and download the latest deployment package. Extract the downloaded files.

Step 2: Start

Run start.bat to launch the program.

Model Loading

Step 1: Download VITS Models

Download the VITS model files and place them in the data/models folder.

Step 2: Loading Models

Automatic Model Loading

Starting from version 0.6.6, it is default behavior to automatically load all models in the data/models folder, making it easier for beginners to use.

Manual Model Loading

After the initial startup, a config.yaml configuration file will be generated. You need to change tts_config.auto_load to false in order to enable manual loading mode.

You can modify the tts_config.models in the config.yaml or make modifications in the admin panel in the browser.

Note: After version 0.6.6, the model loading path has been modified. Please follow the steps below to configure the model path again!

The path can be an absolute path or a relative path. If it's a relative path, it starts from the data/models folder in the project root directory.

For example, if the data/models folder has the following files:

├─model1
│  │─G_1000.pth
│  └─config.json
└─model2
   │─G_1000.pth
   └─config.json

Fill in the configuration like this in the YAML file:

tts_config:
  auto_load: false
  models:
  - config_path: model1/config.json
    model_path: model1/G_1000.pth
  - config_path: model2/config.json
    model_path: model2/G_1000.pth
	# GPT-SoVITS
  - sovits_path: gpt_sovits1/model1_e8_s11536.pth
    gpt_path: gpt_sovits1/model1-e15.ckpt
  - sovits_path: gpt_sovits2/model2_e8_s11536.pth
    gpt_path: gpt_sovits2/model2-e15.ckpt

Loading models through the admin panel is convenient, but if you want to load models outside the data/models folder, you can only do so by modifying the config.yaml configuration file. The method is to directly provide the absolute path.

Absolute path example:

tts_config:
  auto_load: false
  models:
  - config_path: D://model3/config.json
    model_path: D://model3/G_1000.pth

• models_path: This is the models folder relative to the data directory, with the default value being "models". When auto_load is set to true, all models in the models_path directory will be loaded.

Other Models

After downloading the BERT model and emotion model, place them in the data/bert and data/emotional folders respectively. Find the corresponding names and insert them accordingly.

GPU accelerated

Windows

Install CUDA

Check the highest version of CUDA supported by your graphics card:

nvidia-smi

Taking CUDA 11.7 as an example, download it from the official website

Install GPU version of PyTorch

https://pytorch.org/

pip install torch --index-url https://download.pytorch.org/whl/cu118

Linux

The installation process is similar, but I don't have the environment to test it.

WebUI

Inference Frontend

http://127.0.0.1:23456

*Port is modifiable under the default setting of port 23456.

Admin Backend

The default address is http://127.0.0.1:23456/admin.

The initial username and password can be found by searching for 'admin' in the config.yaml file after the first startup.

Function Options Explanation

Disable the Admin Backend

The admin backend allows loading and unloading models, and while it has login authentication, for added security, you can disable the admin backend in the config.yaml:

'IS_ADMIN_ENABLED': !!bool 'false'

This extra measure helps ensure absolute security when making the admin backend inaccessible to the public network.

Bert-VITS2 Configuration and Language/Bert Model Usage

Starting from Bert-VITS2 v2.0, a model requires loading three different language Bert models. If you only need to use one or two languages, you can add the lang parameter in the config.json file of the model's data section. The value ["zh"] indicates that the model only uses Chinese and will load Chinese Bert models. The value ["zh", "ja"] indicates the usage of both Chinese and Japanese bilingual models, and only Chinese and Japanese Bert models will be loaded. Similarly, this pattern continues for other language combinations.

Example:

"data": {
  "lang": ["zh", "ja"],
  "training_files": "filelists/train.list",
  "validation_files": "filelists/val.list",
  "max_wav_value": 32768.0,
  ...

Custom Chinese Polyphonic Dictionary

If you encounter issues with incorrect pronunciation of polyphonic characters, you can try resolving it using the following method.

Create and open phrases_dict.txt in the data directory to add polyphonic words.

{
"一骑当千": [["yí"], ["jì"], ["dāng"], ["qiān"]],
}

GPT-SoVITS Reference Audio Presets

Find the configuration for GPT-SoVITS in the config.yaml file. Add presets under the presets section. Multiple presets can be added, with keys serving as preset names. Below are two default presets, default and default2:

gpt_sovits_config:
  hz: 50
  is_half: false
  id: 0
  lang: auto
  format: wav
  segment_size: 50
  presets:
    default:
      refer_wav_path: null
      prompt_text: null
      prompt_lang: auto
    default2:
      refer_wav_path: null
      prompt_text: null
      prompt_lang: auto

Reading API

Tested in legado

Multiple models can be used for reading, including VITS, Bert-VITS2, GPT-SoVITS. Parameters starting with in configure the speaker of the text in quotes, while parameters starting with nr configure the narrator.

To use GPT-SoVITS, it is necessary to configure the reference audio in the presets section of the config.yaml file in advance and modify the preset in the URL below.

The IP in the URL can be found after the API is started, generally using a local area network IP starting with 192.168.

After modification, select the reading engine, add the reading engine, paste the source, and enable the reading engine.

{
  "concurrentRate": "1",
  "contentType": "audio/wav",
  "enabledCookieJar": false,
  "header": "",
  "id": 1709643305070,
  "lastUpdateTime": 1709821070082,
  "loginCheckJs": "",
  "loginUi": "",
  "loginUrl": "",
  "name": "vits-simple-api",
  "url": "http://192.168.xxx.xxx:23456/voice/reading?text={{java.encodeURI(speakText)}}&in_model_type=GPT-SOVITS&in_id=0&in_preset=default&nr_model_type=BERT-VITS2&nr_id=0&nr_preset=default&format=wav&lang=zh"
}

Frequently Asked Questions

Bert-VITS2 Version Compatibility

To ensure compatibility with the Bert-VITS2 model, modify the config.json file by adding a version parameter "version": "x.x.x". For instance, if the model version is 1.0.1, the configuration file should be written as:

{
  "version": "1.0.1",
  "train": {
    "log_interval": 10,
    "eval_interval": 100,
    "seed": 52,
    ...

Please note that for the Chinese extra version, the version should be changed to extra or zh-clap, and for the extra fix version, the version should be 2.4 or extra-fix.

API

GET

speakers list

• GET http://127.0.0.1:23456/voice/speakers

  Returns the mapping table of role IDs to speaker names.

voice vits

• GET http://127.0.0.1:23456/voice/vits?text=text

  Default values are used when other parameters are not specified.

• GET http://127.0.0.1:23456/voice/vits?text=[ZH]text[ZH][JA]text[JA]&lang=mix

  When lang=mix, the text needs to be annotated.

• GET http://127.0.0.1:23456/voice/vits?text=text&id=142&format=wav&lang=zh&length=1.4

  The text is "text", the role ID is 142, the audio format is wav, the text language is zh, the speech length is 1.4, and the other parameters are default.

check

• GET http://127.0.0.1:23456/voice/check?id=0&model=vits

POST

• See api_test.py

API KEY

Set api_key_enabled: true in config.yaml to enable API key authentication. The API key is api_key: api-key. After enabling it, you need to add the api_key parameter in GET requests and add the X-API-KEY parameter in the header for POST requests.

Parameter

VITS

VITS voice conversion

HuBert-VITS

W2V2-VITS

Dimensional emotion

Bert-VITS2

GPT-SoVITS Speech Synthesis

SSML (Speech Synthesis Markup Language)

Supported Elements and Attributes

speak Element

voice Element

Higher priority than speak.

break Element

Reading

The other parameters of the model will use the default parameters of the corresponding model in the config.yaml file.

Example

See api_test.py

Communication

Learning and communication,now there is only Chinese QQ group

Acknowledgements

• vits:https://github.com/jaywalnut310/vits
• MoeGoe:https://github.com/CjangCjengh/MoeGoe
• emotional-vits:https://github.com/innnky/emotional-vits
• vits-uma-genshin-honkai:https://huggingface.co/spaces/zomehwh/vits-uma-genshin-honkai
• vits_chinese:https://github.com/PlayVoice/vits_chinese
• Bert_VITS2:https://github.com/fishaudio/Bert-VITS2
• GPT-SoVITS:https://github.com/RVC-Boss/GPT-SoVITS

Thank You to All Contributors