Synthesizing test sentences warnings during training

[mbarnig]

When training a model, some common warnings are shown below:

1. amplitude out of range

 | > Synthesizing test sentences.
warning: audio amplitude out of range, auto clipped.
warning: audio amplitude out of range, auto clipped.
warning: audio amplitude out of range, auto clipped.
warning: audio amplitude out of range, auto clipped.
warning: audio amplitude out of range, auto clipped.
warning: audio amplitude out of range, auto clipped.

2. Decoder stopped

 | > Synthesizing test sentences.
   > Decoder stopped with `max_decoder_steps` 1000
   > Decoder stopped with `max_decoder_steps` 1000
   > Decoder stopped with `max_decoder_steps` 1000
   > Decoder stopped with `max_decoder_steps` 1000
   > Decoder stopped with `max_decoder_steps` 1000

3. User warning

 | > Synthesizing test sentences.
/home/mbarnig/COQUI-TTS/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=1024 is too small for input signal of length=768   

I usually ignore these warnings if the loss is decreasing as expected, but I'm not shure if this is the best reaction !
Are there some recommendations how to avoid these warnings ?

[skol101]

When training a model, some common warnings are shown below:

1. amplitude out of range

 | > Synthesizing test sentences.
warning: audio amplitude out of range, auto clipped.
warning: audio amplitude out of range, auto clipped.
warning: audio amplitude out of range, auto clipped.
warning: audio amplitude out of range, auto clipped.
warning: audio amplitude out of range, auto clipped.
warning: audio amplitude out of range, auto clipped.

Had same warnings when was training FastPitch.

[thorstenMueller]

Hi @mbarnig, as discussed in our call now maybe playing around with test_delay_epochs parameter might reduce errors that occur from too early state of training.

[mbarnig]

Referring to the answers given in the first Coqui-TTS online meeting and to the contribution of @thorstenMueller, my understanding concerning the warnings about out of range values is the following :

At the beginning of the training these warnings are normal. The warnings should disappear after some hundred epochs. If the values are still out of range when the model is converging, there is an issue with the dataset or with the model parameters, which must be solved before restarting or continuing the training.

The output of the warnings can be disabled up to a threshold with the parameter test_delay_epochs to reduce the size of the logs.

The value of the max_decoder_steps can be increased during inference to allow the synthesis of longer sentences. Increasing this value during training is not recommended, because it needs more memory and risks to crash the training process.