text_to_speech.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text-To-Speech inference\n",
    "\n",
    "This notebook illustrates how to use the `tts` function to automatically convert text to natural speech ! The function leverages the `Tacotron2` (with `SV2TTS` variant) as `synthesizer`, and `WaveGlow` as `vocoder` to produce natural speech :)\n",
    "\n",
    "Note that all models are loaded as `singleton`, meaning that only the 1st call will be slow (due to model loading + compilation), and the subsequent calls will be **much faster**, up to 10 times real-time on RTX3090 ! (i.e., it takes 1 second to generate a 10 seconds audio)\n",
    "\n",
    "All the audio outputs are available in the `example_outputs` directory, so that you can listen to them without loading the models :)\n",
    "\n",
    "PS : the last French model demo will not be shared, but it is a good example of `SV2TTS` fine-tuning on single-speakers with only limited amount of good quality data :) Nonetheless, the foundation model (i.e. multi-speaker version), and the training code will be shared so that you can replicate this on your own data !\n",
    "\n",
    "## Steps to reproduce\n",
    "\n",
    "1. Download model weights (see `README.md` for the links)\n",
    "2. Unzip weights in `pretrained_models/` directory\n",
    "3. Execute cells !\n",
    "\n",
    "Note : to associate a model to a language, go to `models/tts/__init__.py` and modify the `_pretrained` global variable (at the end of the file). The `key` is the language and the `value` is de model's name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Streaming\n",
    "\n",
    "This functionality allows to enter text and *directly* (up to the inference time) get the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from models.tts import tts_stream\n",
    "\n",
    "# I suggest you to not save audios when streaming because it is slower ;)\n",
    "# I did it so that you can listen to the result\n",
    "\n",
    "# PS : the 'goodbye, see you soon !' message at the end is a funny sentence I added\n",
    "# when you stop the streaming :D\n",
    "tts_stream(model = 'pretrained_tacotron2', directory = 'example_outputs/streaming', overwrite = True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TTS on text\n",
    "\n",
    "This function generates audios based on the provided (list of) text(s). \n",
    "\n",
    "By default, the model does not re-generate sentences if they have already been generated. This behavior can modified by passing the `overwrite = True` argument, to force regeneration. Note that for `SV2TTS`-based models, `overwrite` is `True` by default, as those models are designed to have multiple intonations based on the input embeddings. \n",
    "\n",
    "In this example, the model is loaded with `lang = 'en'`, which loads the `pretrained_tacotron2` model. In the `models/tts/__init__.py` file, it is possible to modify the association by changing the global `_pretrained` variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from models.tts import tts\n",
    "from loggers import set_level\n",
    "\n",
    "text = \"\"\"\n",
    "Hello world ! I hope you will enjoy this funny API for Text-To-Speech ! \n",
    "\"\"\"\n",
    "\n",
    "set_level('info')\n",
    "\n",
    "_ = tts(\n",
    "    text, lang = 'en', directory = 'example_outputs/en', display = True, overwrite = True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from models.tts import tts\n",
    "\n",
    "text = [\n",
    "    \"Bonjour tout le monde ! J'espère que vous allez aimer cette démonstration de voix en français !\"\n",
    "]\n",
    "\n",
    "_ = tts(\n",
    "    text, lang = 'fr', directory = 'example_outputs/fr', display = True,  overwrite = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In regular inference, `Tacotron-2` (and `SV2TTS`) models struggle to predict long texts (e.g., longer than ~150 caracters), due to their attention mechanism. To mitigate this limitation, the `attn_mask_win_len` is introduced to dynamically move a sliding window to keep at most the given number of tokens visible by the attention. It is therefore important to correctly set this parameter when your text is too long, or use the `max_text_length` argument to split the text.\n",
    "The 2nd solution works well, but gives results of lower quality as the model do not have access to the entire text, and will therefore not produce a *smooth and continuous* reading of the text. Nonetheless, the masking feature is still experimental, and may fail in some cases ;) Thanks to the randomness nature of the model, a simple workaround is to re-execute the generation (with `overwrite = True`) to have another result !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from models.tts import tts\n",
    "\n",
    "text = \"\"\"\n",
    "Bonjour tout le monde ! J'espère que vous allez aimer cette démonstration de mon super modèle entrainé avec seulement 20 minutes \n",
    "d'audios ! Je ne partagerai pas ce modèle, mais je trouvais ça intéressant de montrer ce qu'il était possible de faire !\n",
    "\"\"\"\n",
    "\n",
    "_ = tts(\n",
    "    text, model = 'sv2tts_fine_tuned', directory = 'example_outputs/fr', attn_mask_win_len = 150, display = True, overwrite = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell displays an audio based on its saving directory and the text. The text is saved in its *cleaned* form, meaning that this original (uncleaned) text will not match any saved audio. It is the reason why the `model` argument is added, in order to load its `TextEncoder`, cleans the text and finally gets it. The `add_model_name` is also required to not add the model's name at the end of the directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from models.tts import get_audio_file\n",
    "from utils.audio import display_audio\n",
    "\n",
    "text = \"\"\"\n",
    "Bonjour tout le monde ! J'espère que vous allez aimer cette démonstration de mon super modèle entrainé avec seulement 20 minutes \n",
    "d'audios ! Je ne partagerai pas ce modèle, mais je trouvais ça intéressant de montrer ce qu'il était possible de faire !\n",
    "\"\"\"\n",
    "\n",
    "_ = display_audio(get_audio_file(text, directory = 'example_outputs/fr', model = 'sv2tts_siwis_v3', add_model_name = False))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}