GST Prosody transfer on Tacotron 2 is not working

[saibharani]

Describe the bug

I trained an Tacotron2 GST model on LJspeech dataset and own Emotional dataset for 100k steps using use_gst=True, gst=GSTConfig(), options in training.

To Reproduce

But during inference the audio sounds same with or without using style_wav. I used from TTS.utils.synthesizer import Synthesizer function for synthesizing the text. Can you suggest any changes in config or any corrections to improve the prosody transfer quality.

Expected behavior

No response

Logs

No response

Environment

- TTS version 0.7.0
- Pytorch version 1.9.0
- CUDA version 10.2
- OS Ubuntu 18.04
- Python 3.8.13

Additional context

Any help is appreciated, thanks.

[WeberJulian]

Hey, LJSpeech might not be the best dataset to train GST on because of the lack of prosody. In the paper they use data from the 2013 Blizzard Challenge. But you might try with audio book data as well since it tends to be more expressive.

[WeberJulian]

You can also try inference by tweaking manually the style tokens to see if you can get variations.

[saibharani]

That makes sense I will try to train a new model using Blizzard dataset or any Audio book data and will give an update here.

Can you please elaborate on tweaking style tokens manually, do you mean by inputting a dict to gst_style_input_weights parameter? can you give any example of manual style tokens?

Thanks

[erogol]

Yes, you can try different weights and see how the output changes. There is no recipe for the right weights.

[erogol]

I move it as it is not a functional issue.

[WeberJulian]

Hey @saibharani, did you get better results with another dataset?

[saibharani]

And can these ForwardTTS models be made MultiLingual? If yes can you point me to any example implementations.

[WeberJulian]

I don't think it has been tested yet, but it shouldn't be too hard to do.

[saibharani]

I was referring to the LanguageEmbedding layer in VITS can it be implemented in ForwardTTS?

[saibharani]

and can we add new languages once the LanguageEmbedding layer is trained in VITS? I have a model in VITS and I want to add new Languages to it

[WeberJulian]

Yeah but you have to do it manually.

[jaggzh]

Can anyone post example code for tacotron2 + gst training and inference? (Possibly with the style tokens provided during inference, but I can probably get that in there myself.)