🚀AI拟声: 5秒内克隆您的声音并生成任意语音内容 Clone a voice in 5 seconds to generate arbitrary speech in real-time
Go to file
2021-08-28 19:56:19 +08:00
archived_untest_files hide untest to avoid confusion when using this repro 2021-08-11 23:37:06 +08:00
encoder Built-in pretrained encoder/vocoder model 简化配置流程,预集成模型 2021-08-16 22:18:46 +08:00
samples Init to support Chinese Dataset. 2021-08-07 11:56:00 +08:00
synthesizer [dataset]support aishell3(tested) 2021-08-25 23:11:29 +08:00
toolbox 【bugfix】 fix bug causing non-sense output for long texts 修复多段文字发音错误 2021-08-22 23:44:25 +08:00
utils Init to support Chinese Dataset. 2021-08-07 11:56:00 +08:00
vocoder Built-in pretrained encoder/vocoder model 简化配置流程,预集成模型 2021-08-16 22:18:46 +08:00
.gitattributes Init to support Chinese Dataset. 2021-08-07 11:56:00 +08:00
.gitignore Built-in pretrained encoder/vocoder model 简化配置流程,预集成模型 2021-08-16 22:18:46 +08:00
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md 2021-08-18 16:37:25 +08:00
demo_toolbox.py Init to support Chinese Dataset. 2021-08-07 11:56:00 +08:00
LICENSE.txt Init to support Chinese Dataset. 2021-08-07 11:56:00 +08:00
README-CN.md Update README-CN.md 2021-08-28 19:55:42 +08:00
README.md Update README.md 2021-08-28 19:56:19 +08:00
requirements.txt Init to support Chinese Dataset. 2021-08-07 11:56:00 +08:00
synthesizer_preprocess_audio.py [dataset]support aishell3(tested) 2021-08-25 23:11:29 +08:00
synthesizer_preprocess_embeds.py Init to support Chinese Dataset. 2021-08-07 11:56:00 +08:00
synthesizer_train.py Init to support Chinese Dataset. 2021-08-07 11:56:00 +08:00

mockingbird

MIT License

This repository is forked from Real-Time-Voice-Cloning which only support English.

English | 中文

Features

🌍 Chinese supported mandarin and tested with multiple datasets: aidatatang_200zh, magicdata, aishell3

🤩 PyTorch worked for pytorch, tested in version of 1.9.0(latest in August 2021), with GPU Tesla T4 and GTX 2060

🌍 Windows + Linux tested in both Windows OS and linux OS after fixing nits

🤩 Easy & Awesome effect with only newly-trained synthesizer, by reusing the pretrained encoder/vocoder

DEMO VIDEO

Quick Start

1. Install Requirements

Follow the original repo to test if you got all environment ready. **Python 3.7 or higher ** is needed to run the toolbox.

If you get an ERROR: Could not find a version that satisfies the requirement torch==1.9.0+cu102 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2 ) This error is probably due to a low version of python, try using 3.9 and it will install successfully

  • Install ffmpeg.
  • Run pip install -r requirements.txt to install the remaining necessary packages.
  • Install webrtcvad pip install webrtcvad-wheels(If you need)

Note that we are using the pretrained encoder/vocoder but synthesizer, since the original model is incompatible with the Chinese sympols. It means the demo_cli is not working at this moment.

2. Train synthesizer with your dataset

  • Download aidatatang_200zh or other dataset and unzip: make sure you can access all .wav in train folder
  • Preprocess with the audios and the mel spectrograms: python synthesizer_preprocess_audio.py <datasets_root> Allow parameter --dataset {dataset} to support adatatang_200zh, magicdata, aishell3

If it happens the page file is too small to complete the operation, please refer to this video and change the virtual memory to 100G (102400), for example : When the file is placed in the D disk, the virtual memory of the D disk is changed.

  • Preprocess the embeddings: python synthesizer_preprocess_embeds.py <datasets_root>/SV2TTS/synthesizer

  • Train the synthesizer: python synthesizer_train.py mandarin <datasets_root>/SV2TTS/synthesizer

  • Go to next step when you see attention line show and loss meet your need in training folder synthesizer/saved_models/.

FYI, my attention came after 18k steps and loss became lower than 0.4 after 50k steps. attention_step_20500_sample_1 step-135500-mel-spectrogram_sample_1

2.2 Use pretrained model of synthesizer

Thanks to the community, some models will be shared:

author Download link Previow Video
@miven https://pan.baidu.com/s/1PI-hM3sn5wbeChRryX-RCQ code2021 https://www.bilibili.com/video/BV1uh411B7AD/

A link to my early trained model: Baidu Yun Codeaid4

3. Launch the Toolbox

You can then try the toolbox:

python demo_toolbox.py -d <datasets_root> or python demo_toolbox.py

Good news🤩: Chinese Characters are supported

TODO

  • Add demo video
  • Add support for more dataset
  • Upload pretrained model
  • Support parallel tacotron
  • Service orianted and docterize
  • 🙏 Welcome to add more