{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "MongolianTTS.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YmNSTzuGFiDj",
        "colab_type": "text"
      },
      "source": [
        "# Mongolian Text To Speech\n",
        "\n",
        "This is an open source Mongolian text to speech implementing the paper:\n",
        "```\n",
        "Hideyuki Tachibana, Katsuya Uenoyama, Shunsuke Aihara\n",
        "Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention\n",
        "https://arxiv.org/abs/1710.08969\n",
        "```\n",
        "\n",
        "The repo containing the implementation can be found here: [https://github.com/tugstugi/pytorch-dc-tts](https://github.com/tugstugi/pytorch-dc-tts). The Mongolian Bible audio book is used as the training dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N4vqXzZI2fro",
        "colab_type": "text"
      },
      "source": [
        "## Setup\n",
        "\n",
        "### Install dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G4-CKFr22uBB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "from os.path import exists, join, expanduser\n",
        "\n",
        "project_name = \"pytorch-dc-tts\"\n",
        "if not exists(project_name):\n",
        "  ! git clone --quiet https://github.com/tugstugi/{project_name}\n",
        "  ! cd {project_name} && pip install -q -r requirements.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n5XKzALS3P1r",
        "colab_type": "text"
      },
      "source": [
        "### Download pretrained models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dc5b0-HV3eh7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# download text2mel\n",
        "if not exists(\"mbspeech-text2mel.pth\"):\n",
        "  ! wget -q -O mbspeech-text2mel.pth https://www.dropbox.com/s/wu26k6tu5hz8hq1/step-200K.pth\n",
        "\n",
        "# download SSRN\n",
        "if not exists(\"mbspeech-ssrn.pth\"):\n",
        "  ! wget -q -O mbspeech-ssrn.pth https://www.dropbox.com/s/tel0xcqa7kkwqze/step-165K.pth"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TiZA7qgn7kCj",
        "colab_type": "text"
      },
      "source": [
        "## Synthesize\n",
        "\n",
        "### Prepare models\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LjqMQn2y6j58",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import sys\n",
        "sys.path.append(project_name)\n",
        "\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")  # ignore warnings in this notebook\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "\n",
        "from tqdm import *\n",
        "import IPython\n",
        "from IPython.display import Audio\n",
        "\n",
        "from hparams import HParams as hp\n",
        "from audio import save_to_wav\n",
        "from models import Text2Mel, SSRN\n",
        "from datasets.mb_speech import vocab, idx2char, get_test_data, number2word"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eVxfJD6I7yca",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "torch.set_grad_enabled(False)\n",
        "text2mel = Text2Mel(vocab)\n",
        "text2mel.load_state_dict(torch.load(\"mbspeech-text2mel.pth\").state_dict())\n",
        "text2mel = text2mel.eval()\n",
        "ssrn = SSRN()\n",
        "ssrn.load_state_dict(torch.load(\"mbspeech-ssrn.pth\").state_dict())\n",
        "ssrn = ssrn.eval()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h_VB9q8q_Wq1",
        "colab_type": "text"
      },
      "source": [
        "### Allowed characters\n",
        "\n",
        "абвгдеёжзийклмноөпрстуүфхцчшъыьэюя-.,!?\n",
        "\n",
        "### Sentences to synthesize"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GtgxZbfG_DgM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "SENTENCES = [\n",
        "    \"Хэнтий, Хангай, Соёны өндөр сайхан нуруунууд. Хойд зүгийн чимэг болсон ой хөвч уулнууд.\",\n",
        "    \"Мэнэн, Шарга, Номины өргөн их говиуд. Өмнө зүгийн манлай болсон элсэн манхан далайнууд.\", \n",
        "    \"Энэ бол миний төрсөн нутаг. Монголын сайхан орон.\"\n",
        "]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yT7GG7OA_0Tp",
        "colab_type": "text"
      },
      "source": [
        "### Synthetize on CPU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jLU2p4Gq_12d",
        "colab_type": "code",
        "outputId": "4af92580-0186-46a1-d5f6-aecd4ab846f1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 246
        }
      },
      "source": [
        "# synthetize by one by one because there is a batch processing bug!\n",
        "for i in range(len(SENTENCES)):\n",
        "    sentence = ' '.join([number2word(s) if s.isdigit() else s for s in SENTENCES[i].split()])\n",
        "    normalized_sentence = \"\".join([c if c.lower() in vocab else '' for c in sentence])\n",
        "    print(normalized_sentence)\n",
        "    \n",
        "    sentences = [normalized_sentence]\n",
        "    max_N = len(normalized_sentence)\n",
        "    L = torch.from_numpy(get_test_data(sentences, max_N))\n",
        "    zeros = torch.from_numpy(np.zeros((1, hp.n_mels, 1), np.float32))\n",
        "    Y = zeros\n",
        "    A = None\n",
        "\n",
        "    for t in range(hp.max_T):\n",
        "      _, Y_t, A = text2mel(L, Y, monotonic_attention=True)\n",
        "      Y = torch.cat((zeros, Y_t), -1)\n",
        "      _, attention = torch.max(A[0, :, -1], 0)\n",
        "      attention = attention.item()\n",
        "      if L[0, attention] == vocab.index('E'):  # EOS\n",
        "          break\n",
        "\n",
        "    _, Z = ssrn(Y)\n",
        "    \n",
        "    Z = Z.cpu().detach().numpy()\n",
        "    save_to_wav(Z[0, :, :].T, '%d.wav' % (i + 1))\n",
        "    IPython.display.display(Audio('%d.wav' % (i + 1), rate=hp.sr))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Хэнтий, Хангай, Соёны өндөр сайхан нуруунууд. Хойд зүгийн чимэг болсон ой хөвч уулнууд.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 786512 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Мэнэн, Шарга, Номины өргөн их говиуд. Өмнө зүгийн манлай болсон элсэн манхан далайнууд.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 753744 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Энэ бол миний төрсөн нутаг. Монголын сайхан орон.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "                <audio controls=\"controls\" >\n",
              "                    <source src=\"data:audio/x-wav;base64,[encoded data, 442448 characters]\" type=\"audio/x-wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    }
  ]
}