StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models

Yinghao Aaron Li, Cong Han, Vinay S. Raghavan, Gavin Mischler, Nima Mesgarani

In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS synthesis on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs.

Paper: https://arxiv.org/abs/2306.07691

Audio samples: https://styletts2.github.io/

Hello redditors,

Recently I've been playing with Style TTS 2 and I have to say the inference speed versus quality is quite good. It's fast and quality is not bad by any means.

For example, the inference of pre-trained LJ speech model is great. Although the quality of the speech isn't the best, but the intonation, pauses and everything else is quite natural, if not for the LJ-speech dataset quality itself. I think it would be great.

I have very old video card only 4GB and I am still able to inference quite a bit of text in not such a long time. It is impressive for sure.

curious for anyone who pre-trained their own models with this what is your opinion?

I'm posting here not only to get the opinion from people who used it, but also to ask if anyone is willing to share their pre-trained model with me. I'm gonna give you two reasons below why I need this. And I would absolutely appreciate anyone's help in this matter.

1 I am blind and I desperately need more natural text to speech system other than SAPI on windows or or standard text to speech output on iOS. telling you folks, using such systems is demotivating to read anything.

2 I don't have a budget to buy RTX 4090 GPU or a skills just yet to pre-traine my own model.

11 labs is definitely too expensive to convert longer text. Let's say a textbook to audio. That's for damn sure. play.ht isn't cheap either. I suppose I could pay 99 dollars or so for unlimited conversions. But that isn't feasible either for me.

tortoise-tts is way too computationally expencive for any text to audiobook making procedures that for sure.

then I thought about RVC but for that you also need a decent TTS solution and from my testing I think if I have good enough pre-trained model for StyleTTS I could experiment further with RVC if needed.

Yeah that's my thoughts if anyone is willing to help me out DM me because I suppose nobody wants to share their models publikly.

I perfectly understand the issues surrounding sharing pre-trained models or audio. So I can promise 3 things for anyone who is willing to help in my situation.

1 I will never share your model with anybody.

2 I will never share the audio generated with your given model publicly.

3 It will be used for my reading activities because that's my intention.

I perfectly understand that the post title is a bit of a clickbait, I suppose, but I want people to actually read the post and asking for help in a title is discouraging. So sorry for that...

I appreciate any comments and opinions, particularly from the people who can evaluate the style TTS 2 performance over the other available options, because that is above my pay grade and knowledge to evaluate how good it is in comparison to other implementations, particularly where diffusion is concerned...