{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "EnglishTTS.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YmNSTzuGFiDj",
        "colab_type": "text"
      },
      "source": [
        "# English Text To Speech\n",
        "\n",
        "This is an open source English text to speech implementing the paper:\n",
        "```\n",
        "Hideyuki Tachibana, Katsuya Uenoyama, Shunsuke Aihara\n",
        "Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention\n",
        "https://arxiv.org/abs/1710.08969\n",
        "```\n",
        "\n",
        "The repo containing the implementation can be found here: [https://github.com/tugstugi/pytorch-dc-tts](https://github.com/tugstugi/pytorch-dc-tts). The [LJ-Speech](https://keithito.com/LJ-Speech-Dataset/) is used as the training dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N4vqXzZI2fro",
        "colab_type": "text"
      },
      "source": [
        "## Setup\n",
        "\n",
        "### Install dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G4-CKFr22uBB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "from os.path import exists, join, expanduser\n",
        "\n",
        "project_name = \"pytorch-dc-tts\"\n",
        "if not exists(project_name):\n",
        "  ! git clone --quiet https://github.com/tugstugi/{project_name}\n",
        "  ! cd {project_name} && pip install -q -r requirements.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n5XKzALS3P1r",
        "colab_type": "text"
      },
      "source": [
        "### Download pretrained models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dc5b0-HV3eh7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# download text2mel\n",
        "if not exists(\"ljspeech-text2mel.pth\"):\n",
        "  ! wget -q -O ljspeech-text2mel.pth https://www.dropbox.com/s/4t13ugxzzgnocbj/step-300K.pth\n",
        "\n",
        "# download SSRN\n",
        "if not exists(\"ljspeech-ssrn.pth\"):\n",
        "  ! wget -q -O ljspeech-ssrn.pth https://www.dropbox.com/s/gw4aqrgcvccmg0g/step-100K.pth"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TiZA7qgn7kCj",
        "colab_type": "text"
      },
      "source": [
        "## Synthesize\n",
        "\n",
        "### Prepare models\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LjqMQn2y6j58",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import sys\n",
        "sys.path.append(project_name)\n",
        "\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")  # ignore warnings in this notebook\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "\n",
        "from tqdm import *\n",
        "import IPython\n",
        "from IPython.display import Audio\n",
        "\n",
        "from hparams import HParams as hp\n",
        "from audio import save_to_wav\n",
        "from models import Text2Mel, SSRN\n",
        "from datasets.lj_speech import vocab, idx2char, get_test_data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eVxfJD6I7yca",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "torch.set_grad_enabled(False)\n",
        "text2mel = Text2Mel(vocab)\n",
        "text2mel.load_state_dict(torch.load(\"ljspeech-text2mel.pth\").state_dict())\n",
        "text2mel = text2mel.eval()\n",
        "ssrn = SSRN()\n",
        "ssrn.load_state_dict(torch.load(\"ljspeech-ssrn.pth\").state_dict())\n",
        "ssrn = ssrn.eval()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h_VB9q8q_Wq1",
        "colab_type": "text"
      },
      "source": [
        "### Allowed characters\n",
        "\n",
        "abcdefghijklmnopqrstuvwxyz'.?\n",
        "\n",
        "### Sentences to synthesize"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GtgxZbfG_DgM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "SENTENCES = [\n",
        "    \"The birch canoe slid on the smooth planks.\",\n",
        "    \"Glue the sheet to the dark blue background.\",\n",
        "    \"It's easy to tell the depth of a well.\",\n",
        "    \"These days a chicken leg is a rare dish.\",\n",
        "    \"Rice is often served in round bowls.\",\n",
        "    \"The juice of lemons makes fine punch.\",\n",
        "    \"The box was thrown beside the parked truck.\",\n",
        "    \"The hogs were fed chopped corn and garbage.\",\n",
        "    \"Four hours of steady work faced us.\",\n",
        "    \"Large size in stockings is hard to sell.\",\n",
        "    \"The boy was there when the sun rose.\",\n",
        "    \"A rod is used to catch pink salmon.\",\n",
        "    \"The source of the huge river is the clear spring.\",\n",
        "    \"Kick the ball straight and follow through.\",\n",
        "    \"Help the woman get back to her feet.\",\n",
        "    \"A pot of tea helps to pass the evening.\",\n",
        "    \"Smoky fires lack flame and heat.\",\n",
        "    \"The soft cushion broke the man's fall.\",\n",
        "    \"The salt breeze came across from the sea.\",\n",
        "    \"The girl at the booth sold fifty bonds.\"\n",
        "]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yT7GG7OA_0Tp",
        "colab_type": "text"
      },
      "source": [
        "### Synthetize on CPU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jLU2p4Gq_12d",
        "colab_type": "code",
        "outputId": "67c50529-6922-4258-9cbd-1717c737e575",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1550
        }
      },
      "source": [
        "# synthetize by one by one because there is a batch processing bug!\n",
        "for i in range(len(SENTENCES)):\n",
        "    sentence = SENTENCES[i]\n",
        "    normalized_sentence = \"\".join([c if c.lower() in vocab else '' for c in sentence])\n",
        "    print(normalized_sentence)\n",
        "    \n",
        "    sentences = [normalized_sentence]\n",
        "    max_N = len(normalized_sentence)\n",
        "    L = torch.from_numpy(get_test_data(sentences, max_N))\n",
        "    zeros = torch.from_numpy(np.zeros((1, hp.n_mels, 1), np.float32))\n",
        "    Y = zeros\n",
        "    A = None\n",
        "\n",
        "    for t in range(hp.max_T):\n",
        "      _, Y_t, A = text2mel(L, Y, monotonic_attention=True)\n",
        "      Y = torch.cat((zeros, Y_t), -1)\n",
        "      _, attention = torch.max(A[0, :, -1], 0)\n",
        "      attention = attention.item()\n",
        "      if L[0, attention] == vocab.index('E'):  # EOS\n",
        "          break\n",
        "\n",
        "    _, Z = ssrn(Y)\n",
        "    \n",
        "    Z = Z.cpu().detach().numpy()\n",
        "    save_to_wav(Z[0, :, :].T, '%d.wav' % (i + 1))\n",
        "    IPython.display.display(Audio('%d.wav' % (i + 1), rate=hp.sr))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The birch canoe slid on the smooth planks.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 311376 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Glue the sheet to the dark blue background.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 300452 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "It's easy to tell the depth of a well.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 251300 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "These days a chicken leg is a rare dish.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 314104 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Rice is often served in round bowls.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 314104 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "The juice of lemons makes fine punch.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 273144 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "The box was thrown beside the parked truck.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 319568 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "The hogs were fed chopped corn and garbage.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 341412 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Four hours of steady work faced us.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 306816 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Large size in stockings is hard to sell.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 341412 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "The boy was there when the sun rose.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 234916 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "A rod is used to catch pink salmon.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 303184 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "The source of the huge river is the clear spring.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 303184 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Kick the ball straight and follow through.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 305912 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Help the woman get back to her feet.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 267684 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "A pot of tea helps to pass the evening.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 297720 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Smoky fires lack flame and heat.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 292260 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "The soft cushion broke the man's fall.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 262224 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "The salt breeze came across from the sea.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 294992 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "The girl at the booth sold fifty bonds.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 303680 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    }
  ]
}