XTTS Dataset Structure Overview & Hyperparameters

[C00reNUT]

Hello,

thank you for modifying tortoise architecture and finetuning it to new languages.

Would it be possible to get a table or an list of how many hours of audio has the model "seen" during training for each specific language?

I am thinking about buying the comercial license and this way I could asses (at least hypothetically) how much better I can finetune the model with my private dataset in my language. I could compare the current performance and see how much data the model has already been trained to and the guess how much better I can make it with my resources.

Also some info about the training and hyperparameters would be really beneficial...

[loganhart02]

These are all the languages right now.
English (en), Spanish (es), French (fr), German (de), Italian (it), Portuguese (pt), Polish (pl), Turkish (tr), Russian (ru), Dutch (nl), Czech (cs), Arabic (ar), Chinese (zh-cn), Japanese (ja), Hungarian (hu) and Korean (ko).
how much data do you have and what language are you interested in adding?

[C00reNUT]

Yes, that is pretty obvious, the question is quantity and quality of the data. I understand that for many reasons you don't publish the dataset. But this makes hard to asses quality and problems with the models that are inherently based on data:

1. I don't know how many hours of audio there are in dataset for each language and therefore I cannot tell how much I can improve the model using finetuning. There are languages in similar language groups, like Slavic (Polish, Czech, Russian) that will probably make the speech better for all of them, at least a bit, even if one is not finetuning for the specific language. Therefore it is beneficial to know the total hours of audio in the dataset for all lanugages.

2. Since you will not publish the dataset I cannot check the quality of captions. Seeing the language range I would bet that you are using whisper for dataset building. I have done the same for tortoise finetuning. The problem with whisper, especially v2, even with published low WER, is that it is quite bad in many languages besides English. I have tried even finetuned versions for specific language and in German an Czech it produces dataset with too many error to be useful for TTS applications, not even close to ljspeech quality.

I have about 5000 captioned hours in English, 2300 in German and 2000 in Czech. I am specially interested in Czech and German data you used to train these models, since for English I can see from tortoise tts paper https://arxiv.org/pdf/2305.07243.pdf that cca 50k hours of english audio was used for training.

Again, I think it would be beneficial for everyone knowing these basic facts about the dataset, like total number of hours of audio in each language and probably how it was processed. This way it is much easier to guess what I can do with my own data, and whether to buy the licence or not.

[Aml-Hassan-Abd-El-hamid]

any update on how many hours were used for each language? I'm especially interested in XTTS v2

[eginhard]

Details are in the paper: https://arxiv.org/abs/2406.04904

[Aml-Hassan-Abd-El-hamid]

Thanks you very much for your help