0.8.0 vits crash when training #1959

neurlang · 2022-09-11T16:45:08Z

neurlang
Sep 11, 2022

On my english dataset I have a problem with rational_quadratic_spline. The input tensor becomes empty.

Traceback (most recent call last):
  File "~/.local/lib/python3.10/site-packages/trainer/trainer.py", line 1533, in fit
    self._fit()
  File "~/.local/lib/python3.10/site-packages/trainer/trainer.py", line 1517, in _fit
    self.train_epoch()
  File "~/.local/lib/python3.10/site-packages/trainer/trainer.py", line 1282, in train_epoch
    _, _ = self.train_step(batch, batch_num_steps, cur_step, loader_start_time)
  File "~/.local/lib/python3.10/site-packages/trainer/trainer.py", line 1135, in train_step
    outputs, loss_dict_new, step_time = self._optimize(
  File "~/.local/lib/python3.10/site-packages/trainer/trainer.py", line 996, in _optimize
    outputs, loss_dict = self._model_train_step(batch, model, criterion, optimizer_idx=optimizer_idx)
  File "~/.local/lib/python3.10/site-packages/trainer/trainer.py", line 954, in _model_train_step
    return model.train_step(*input_args)
  File "~/TTS/TTS/tts/models/vits.py", line 1247, in train_step
    outputs = self.forward(
  File "~/TTS/TTS/tts/models/vits.py", line 1016, in forward
    outputs, attn = self.forward_mas(outputs, z_p, m_p, logs_p, x, x_mask, y_mask, g=g, lang_emb=lang_emb)
  File "~/TTS/TTS/tts/models/vits.py", line 913, in forward_mas
    loss_duration = self.duration_predictor(
  File "~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "~/TTS/TTS/tts/layers/vits/stochastic_duration_predictor.py", line 255, in forward
    z_q, logdet_q = flow(z_q, x_mask, g=(x + h))
  File "~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "~/TTS/TTS/tts/layers/vits/stochastic_duration_predictor.py", line 133, in forward
    x1, logabsdet = piecewise_rational_quadratic_transform(
  File "~/TTS/TTS/tts/layers/vits/transforms.py", line 32, in piecewise_rational_quadratic_transform
    outputs, logabsdet = spline_fn(
  File "~/TTS/TTS/tts/layers/vits/transforms.py", line 80, in unconstrained_rational_quadratic_spline
    outputs[inside_interval_mask], logabsdet[inside_interval_mask] = rational_quadratic_spline(
  File "~/TTS/TTS/tts/layers/vits/transforms.py", line 112, in rational_quadratic_spline
    if torch.min(inputs) < left or torch.max(inputs) > right:
RuntimeError: min(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.

If I workaround, many losses become nan.
It is a problem with dataset because I've trained other languages fine. What problem it could be?

Ca-ressemble-a-du-fake · 2022-09-24T09:16:03Z

Ca-ressemble-a-du-fake
Sep 24, 2022

@neurlang I get the same error all of a sudden after 10k steps with a custom dataset in French with Python 3.8. It has wroked perfectly so far with other datasets also in French.

      --> STEP: 69/91 -- GLOBAL_STEP: 10525
     | > loss_disc: 2.83631  (2.84479)
     | > loss_disc_real_0: 0.25157  (0.22624)
     | > loss_disc_real_1: 0.24458  (0.23003)
     | > loss_disc_real_2: 0.24874  (0.23917)
     | > loss_disc_real_3: 0.24829  (0.24568)
     | > loss_disc_real_4: 0.23296  (0.24652)
     | > loss_disc_real_5: 0.26109  (0.24877)
     | > loss_0: 2.83631  (2.84479)
     | > grad_norm_0: 5.94465  (7.87682)
     | > loss_gen: 1.58478  (1.73315)
     | > loss_kl: 1.31095  (1.22096)
     | > loss_feat: 2.48340  (2.28130)
     | > loss_mel: 13.32878  (12.36625)
     | > loss_duration: 0.85561  (0.97174)
     | > amp_scaler: 64.00000  (64.00000)
     | > loss_1: 19.56352  (18.57340)
     | > grad_norm_1: 127.05968  (335.93912)
     | > current_lr_0: 0.00020
     | > current_lr_1: 0.00020
     | > step_time: 1.35030  (1.16835)
     | > loader_time: 0.02740  (0.01606)

 ! Run is kept in /home/caraduf/VoiceCloning/Models/VITS_RTX-September-23-2022_03+26PM-0000000
Traceback (most recent call last):
  File "/home/caraduf/.local/lib/python3.8/site-packages/trainer/trainer.py", line 1533, in fit
    self._fit()
  File "/home/caraduf/.local/lib/python3.8/site-packages/trainer/trainer.py", line 1517, in _fit
    self.train_epoch() 
  File "/home/caraduf/.local/lib/python3.8/site-packages/trainer/trainer.py", line 1282, in train_epoch
    _, _ = self.train_step(batch, batch_num_steps, cur_step, loader_start_time)
  File "/home/caraduf/.local/lib/python3.8/site-packages/trainer/trainer.py", line 1135, in train_step
    outputs, loss_dict_new, step_time = self._optimize(
  File "/home/caraduf/.local/lib/python3.8/site-packages/trainer/trainer.py", line 996, in _optimize
    outputs, loss_dict = self._model_train_step(batch, model, criterion, optimizer_idx=optimizer_idx)
  File "/home/caraduf/.local/lib/python3.8/site-packages/trainer/trainer.py", line 954, in _model_train_step
    return model.train_step(*input_args)
  File "/home/caraduf/VoiceCloning/YourTTS/TTS/TTS/tts/models/vits.py", line 1247, in train_step
    outputs = self.forward(
  File "/home/caraduf/VoiceCloning/YourTTS/TTS/TTS/tts/models/vits.py", line 1016, in forward
    outputs, attn = self.forward_mas(outputs, z_p, m_p, logs_p, x, x_mask, y_mask, g=g, lang_emb=lang_emb)
  File "/home/caraduf/VoiceCloning/YourTTS/TTS/TTS/tts/models/vits.py", line 913, in forward_mas
    loss_duration = self.duration_predictor(
  File "/home/caraduf/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/caraduf/VoiceCloning/YourTTS/TTS/TTS/tts/layers/vits/stochastic_duration_predictor.py", line 255, in forward
    z_q, logdet_q = flow(z_q, x_mask, g=(x + h))
  File "/home/caraduf/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/caraduf/VoiceCloning/YourTTS/TTS/TTS/tts/layers/vits/stochastic_duration_predictor.py", line 133, in forward
    x1, logabsdet = piecewise_rational_quadratic_transform(
  File "/home/caraduf/VoiceCloning/YourTTS/TTS/TTS/tts/layers/vits/transforms.py", line 32, in piecewise_rational_quadratic_transform
    outputs, logabsdet = spline_fn(
  File "/home/caraduf/VoiceCloning/YourTTS/TTS/TTS/tts/layers/vits/transforms.py", line 80, in unconstrained_rational_quadratic_spline
    outputs[inside_interval_mask], logabsdet[inside_interval_mask] = rational_quadratic_spline(
  File "/home/caraduf/VoiceCloning/YourTTS/TTS/TTS/tts/layers/vits/transforms.py", line 112, in rational_quadratic_spline
    if torch.min(inputs) < left or torch.max(inputs) > right:
RuntimeError: min(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.

Did you find a solution ? Another question suggests to downgrade Pytorch but I am not sure it is useful as it worked well with other (smaller) datasets. The only difference is that this one is longer (5k samples of 1 to 12 s long).

1 reply

Ca-ressemble-a-du-fake Sep 24, 2022

Removing torch (pip3 uninstall torch torchaudio torchvision) and installing the latest version 1.12.1+cu113 swith pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113 did NOT solve the problem.

freezerain · 2022-09-25T13:13:04Z

freezerain
Sep 25, 2022

I got similar problem here
#1949
Is your hardware allow to downgrade to version that worked for me (1.8.0+cu101)?
pip install torch==1.8.0+cu101 torchvision==0.9.0+cu101 torchaudio===0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

If I remember correctly this code:

import torch
print(torch.__version__)

were returning something like "1.12.1+cpu" no matter what was installed, so I force-install bugged version and then downgraded to stable version that I found in SO answers.
Looks like torch bug to me

0 replies

Ca-ressemble-a-du-fake · 2022-09-26T04:17:32Z

Ca-ressemble-a-du-fake
Sep 26, 2022

I tried this method with success. Here are the detailed steps I applied (it sounds like voodoo) :

I upgraded my Ubuntu to the latest one (22.04).
Then after reboot, I created a directory for coqui and created a virtual environment to not mess everything following https://cloud.google.com/python/docs/setup#linux .
And installed coqui tts with pip3 commands following https://tts.readthedocs.io/en/latest/installation.html .
I checked the torch and cuda version with

   import torch 
   print(torch.__version__)
   torch.version.cuda

It returned something with CUDA 10.2 which does not support my hardware (Ampere familly card).

So (still in the early created venv) I removed torch and torchaudio pip3 uninstall torch torchaudio
And installed the stable version (1.12.1) of torch compiled for a Cuda version that supports Ampere GPU (11.6) pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116 following https://pytorch.org/get-started/locally/ .
The training failed because of missing CUDA compiler (can't remember exactly what was missing), so I installed system wise CUDA toolkit 11.7 (even though torch was installed for cuda 11.6) from Nvidia site following their guide for Ubuntu

As a summary drivers have been updated to version 515, CUDA toolkit 11.7, torch stable (1.12.1+cuda11.6).

Now it works as expected.
Hope this helps

3 replies

neurlang Sep 26, 2022
Author

I am not going to risk downgrade partially to cuda 11.6
I am on torch nightly
I'd rather wait for fix in torch nightly if there is a problem in there.

Ca-ressemble-a-du-fake Sep 27, 2022

Otherwise you could try an use an isolated environment (see here) and do the downgrade to Cuda 11.6 in this environment without messing up the main python install.

Ca-ressemble-a-du-fake Sep 29, 2022

Furthermore this is not a bullet proof solution since the problem just reappeared !

erogol · 2022-09-27T09:33:28Z

erogol
Sep 27, 2022
Maintainer

It is in general a sign of unstable training mostly caused by bad samples in your dataset. https://tts.readthedocs.io/en/latest/what_makes_a_good_dataset.html

5 replies

Ca-ressemble-a-du-fake Sep 29, 2022

@erogol Is it possible to know which exact sample or (batch of samples) caused this Error ? I mean which sample was being processed at the time of the error ?

Ca-ressemble-a-du-fake Sep 29, 2022

@erogol For me the problem arose when I added accents in characters list in config.json. This is the only difference with the previous run that was successful (but without accentuated letters). Wav files are identical.

erogol Sep 29, 2022
Maintainer

Do you finetune the prev model with this new char set?

Ca-ressemble-a-du-fake Sep 29, 2022

No no, I started from scratch. I tried to exclude samples shorter than 7 chars but it failed too. I am now trying to exclude samples shorter than 28 chars (around 2 seconds).

Ca-ressemble-a-du-fake Sep 30, 2022

I tried with excluding short samples as explained above but it failed too.

If I try NFKD normalized text (ie "é" will be decomposed into "e" + "◌́" ) without accents in characters list, it now fails at step 6200 with the aforementioned error whereas it worked fine some just a week ago (it reached step 60k).

neurlang · 2022-10-02T07:42:51Z

neurlang
Oct 2, 2022
Author

Same problem on slovak single words dataset.

4 replies

neurlang Oct 2, 2022
Author

same problem on latest dev. I can reproduce it every time.

Ca-ressemble-a-du-fake Oct 2, 2022

@neurlang I reinstalled it via pip install -e .[all] after cloning git clone https://github.com/coqui-ai/TTS. Did I install the dev or the v 0.8 ? Maybe this is the reason why I get a very different behavior.

neurlang Oct 2, 2022
Author

I managed to workaround the problem in sochastic duration predictor:

diff --git a/TTS/tts/layers/vits/stochastic_duration_predictor.py b/TTS/tts/layers/vits/stochastic_duration_predictor.py
index 738ee341..c80e35a9 100644
--- a/TTS/tts/layers/vits/stochastic_duration_predictor.py
+++ b/TTS/tts/layers/vits/stochastic_duration_predictor.py
@@ -119,6 +119,11 @@ class ConvFlow(nn.Module):
 
     def forward(self, x, x_mask, g=None, reverse=False):
         x0, x1 = torch.split(x, [self.half_channels] * 2, 1)
+        if x1[(x1 >= -self.tail_bound) & (x1 <= self.tail_bound)].size()[0] == 0:
+            x = torch.cat([x0, x1], 1) * x_mask
+            if not reverse:
+                return x, None
+            return x
         h = self.pre(x0)
         h = self.convs(h, x_mask, g=g)
         h = self.proj(h) * x_mask
@@ -274,7 +279,8 @@ class StochasticDurationPredictor(nn.Module):
             # flow layers
             for idx, flow in enumerate(flows):
                 z, logdet = flow(z, x_mask, g=x, reverse=reverse)
-                logdet_tot = logdet_tot + logdet
+                if logdet is not None:
+                    logdet_tot = logdet_tot + logdet
                 if idx > 0:
                     z = torch.flip(z, [1])

The problem was that all 16 inputs in the batch happened to be (suspecting a dataset problem) outside the tail bounds -5 to 5 in my case.
I signal using None that it bugged and skip the flow in the flow layers loop.

p0p4k Mar 23, 2023

Also, I had to add these changes, since my error came up at post_flows.

            # posterior encoder
            logdet_tot_q = 0.0
            for idx, flow in enumerate(self.post_flows):
                z_q, logdet_q = flow(z_q, x_mask, g=(x + h))
                # logdet_tot_q = logdet_tot_q + logdet_q
                if logdet_q is not None:
                    logdet_tot_q = logdet_tot_q + logdet_q
                if idx > 0:
                    z_q = torch.flip(z_q, [1])

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

0.8.0 vits crash when training #1959

{{title}}

Replies: 5 comments 13 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

0.8.0 vits crash when training #1959

Replies: 5 comments · 13 replies

neurlang Sep 26, 2022 Author

erogol Sep 27, 2022 Maintainer

erogol Sep 29, 2022 Maintainer

neurlang Oct 2, 2022 Author

neurlang Oct 2, 2022 Author

neurlang Oct 2, 2022 Author

Replies: 5 comments 13 replies

neurlang Sep 26, 2022
Author

erogol
Sep 27, 2022
Maintainer

erogol Sep 29, 2022
Maintainer

neurlang
Oct 2, 2022
Author

neurlang Oct 2, 2022
Author

neurlang Oct 2, 2022
Author