{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "colab": {
      "name": "train-personal-model-with-common-voice.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f79d99ef"
      },
      "source": [
        "# Create a custom Speech-to-Text model for 💫 Your Voice 💫 with Common Voice\n",
        "\n",
        "👋 Hello and welcome to Coqui Speech-to-Text (🐸STT). Check out [🐸STT on Github](https://github.com/coqui-ai/STT), and leave a star for new models and updates ⭐\n",
        "\n",
        "This notebook shows you how to train a **custom, personal model** for **your voice** using **your data** from Common Voice.\n",
        "\n",
        "In this notebook, we will:\n",
        "\n",
        "1. Download your personal Common Voice data\n",
        "2. Import your data into 🐸STT\n",
        "3. Fine-tune an off-the-shelf 🐸STT model to your voice\n",
        "4. Test your custom model and display performance\n",
        "\n",
        "So, let's jump in!"
      ],
      "id": "f79d99ef"
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ⚠️ Caveats ⚠️\n",
        "\n",
        "- *You can only download your data from Common Voice once every seven days, so plan accordingly.*\n",
        "- *You should use a GPU*\n",
        "- *This tutorial assumes your Common Voice data is in English. If you'd like to fine-tune a STT model in another language, you will need to download the appropriate pre-trained model checkpoints.*\n",
        "- *This tutorial assumes you've recorded at least 100 sentences in Common Voice. If you've recorded more or less you should change the batch sizes. The more you record, the better.*"
      ],
      "metadata": {
        "id": "IAA-f0X0JJM7"
      },
      "id": "IAA-f0X0JJM7"
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ✅ Make sure you're using a GPU\n",
        "\n",
        "On Colab you should have free GPU access, so let's use the power 🔥\n",
        "\n",
        "Turn on the GPU runtime from the Google Colab menu as follows:\n",
        "\n",
        "`Runtime >> Change runtime type >> Hardware Accelerator >> GPU`\n"
      ],
      "metadata": {
        "id": "yYxr-XtzHD2q"
      },
      "id": "yYxr-XtzHD2q"
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ✅ Download your data from Common Voice\n",
        "\n",
        "To download data from Common Voice, you need to have a Common Voice account and you must have recorded some sentences. Log-in or Sign-up in the top right corner of [`commonvoice.mozilla.org`](https://commonvoice.mozilla.org)\n",
        "\n",
        "Once you've logged in, you will see your username and avatar in the top right corner.\n",
        "\n",
        "From the drop-down menu next to your username, click on \"Profile\"."
      ],
      "metadata": {
        "id": "0psrUINITsbC"
      },
      "id": "0psrUINITsbC"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "be5fe49c"
      },
      "source": [
        "\n",
        "![profile-dropdown.jpg](data:image/jpeg;base64,[encoded data, 65544 characters])\n"
      ],
      "id": "be5fe49c"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, click the \"Download My Data\" button."
      ],
      "metadata": {
        "id": "7kQqjxOTTlY5"
      },
      "id": "7kQqjxOTTlY5"
    },
    {
      "cell_type": "markdown",
      "source": [
        "![download-data.jpg](data:image/jpeg;base64,[encoded data, 66492 characters])"
      ],
      "metadata": {
        "id": "5UrFTOoaB_ec"
      },
      "id": "5UrFTOoaB_ec"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, push the \"Request recordings\" button."
      ],
      "metadata": {
        "id": "2VqqndcjTcYW"
      },
      "id": "2VqqndcjTcYW"
    },
    {
      "cell_type": "markdown",
      "source": [
        "![Screen Shot 2022-01-10 at 8.01.06 PM.jpg](data:image/jpeg;base64,[encoded data, 162324 characters])"
      ],
      "metadata": {
        "id": "JLTCkVzlUart"
      },
      "id": "JLTCkVzlUart"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, click the \"Download\" button"
      ],
      "metadata": {
        "id": "fxIzQwjpWGIZ"
      },
      "id": "fxIzQwjpWGIZ"
    },
    {
      "cell_type": "markdown",
      "source": [
        "![Screen Shot 2022-01-10 at 8.10.54 PM.jpg](data:image/jpeg;base64,[encoded data, 72644 characters])"
      ],
      "metadata": {
        "id": "A8128lMTWK5p"
      },
      "id": "A8128lMTWK5p"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally you've made it!\n",
        "\n",
        "You should see a download page with two URLs to your data bundles.\n",
        "\n",
        "Copy and paste those two links into the code below, and let's get started!"
      ],
      "metadata": {
        "id": "_EGX4z0QWTuq"
      },
      "id": "_EGX4z0QWTuq"
    },
    {
      "cell_type": "markdown",
      "source": [
        "![Screen Shot 2022-01-10 at 8.13.03 PM.jpg](data:image/jpeg;base64,[encoded data, 204156 characters])"
      ],
      "metadata": {
        "id": "wV6WWp7cWmGd"
      },
      "id": "wV6WWp7cWmGd"
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ✅ Download your data to Colab\n",
        "\n",
        "Just paste in the URLs you got from Common Voice into the code block below, and run it."
      ],
      "metadata": {
        "id": "imUOwCWsNI_w"
      },
      "id": "imUOwCWsNI_w"
    },
    {
      "cell_type": "code",
      "source": [
        "from urllib import request\n",
        "import os\n",
        "\n",
        "FIRST_URL=\"Paste first super long URL here, but leave in the surrounding double quotes\"\n",
        "SECOND_URL=\"Paste second super long URL here, but leave in the surrounding double quotes\"\n",
        "\n",
        "FIRST_FILENAME = FIRST_URL.split(\"/\")[-1].split(\"?\")[0]\n",
        "SECOND_FILENAME = SECOND_URL.split(\"/\")[-1].split(\"?\")[0]\n",
        "\n",
        "if os.path.isfile(FIRST_FILENAME):\n",
        "  print(\"Found\", FIRST_FILENAME, \", skipping download.\")\n",
        "else:\n",
        "  print(\"Downloading\", FIRST_FILENAME, \"...\")\n",
        "  request.urlretrieve(FIRST_URL, FIRST_FILENAME)\n",
        "\n",
        "if os.path.isfile(SECOND_FILENAME):\n",
        "  print(\"Found\", SECOND_FILENAME, \", skipping download.\")\n",
        "else:\n",
        "  print(\"Downloading\", SECOND_FILENAME, \"...\")\n",
        "  request.urlretrieve(SECOND_URL, SECOND_FILENAME)\n"
      ],
      "metadata": {
        "id": "-CjBXmBSW0vw"
      },
      "id": "-CjBXmBSW0vw",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "96e8b708"
      },
      "source": [
        "### 👀 Take a look at your data"
      ],
      "id": "96e8b708"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fa2aec77"
      },
      "source": [
        "# take a look at what you've downloaded\n",
        "! ls ."
      ],
      "id": "fa2aec77",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ✅ Install 🐸STT"
      ],
      "metadata": {
        "id": "W8KcmpsTX1p2"
      },
      "id": "W8KcmpsTX1p2"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fa2aec78"
      },
      "source": [
        "## Install Coqui STT\n",
        "# dependencies\n",
        "! apt-get install sox libsox-fmt-mp3 libopusfile0 libopus-dev libopusfile-dev\n",
        "! pip install --upgrade pip\n",
        "# the Coqui training package\n",
        "! pip install coqui_stt_training\n",
        "! pip uninstall -y tensorflow; pip install \"tensorflow-gpu==1.15\"\n",
        "# code with importer scripts\n",
        "! git clone --depth=1 https://github.com/coqui-ai/STT.git"
      ],
      "id": "fa2aec78",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ✅ Import your data for 🐸STT\n",
        "\n"
      ],
      "metadata": {
        "id": "6ITXoId-bRIQ"
      },
      "id": "6ITXoId-bRIQ"
    },
    {
      "cell_type": "code",
      "source": [
        "# your *.txt and *.zip files will have different names, these are just examples\n",
        "! python STT/bin/import_cv_personal.py --normalize takeout_*.txt takeout_*.zip\n",
        "\n",
        "# now we're going to split the dataset into {train,dev,test} with auto_import_dataset\n",
        "from coqui_stt_training.util.config import initialize_globals_from_args\n",
        "import glob\n",
        "\n",
        "DATA_CSV=glob.glob(\"/content/takeout_*/data.csv\")[0]\n",
        "\n",
        "initialize_globals_from_args(\n",
        "    auto_input_dataset=DATA_CSV\n",
        ")"
      ],
      "metadata": {
        "id": "Dmm5aIYvbm_q"
      },
      "id": "Dmm5aIYvbm_q",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ✅ Download a pre-trained English 🐸STT model"
      ],
      "metadata": {
        "id": "c8AWzPclhxwu"
      },
      "id": "c8AWzPclhxwu"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "608d203f"
      },
      "outputs": [],
      "source": [
        "### Download pre-trained model\n",
        "import os\n",
        "import tarfile\n",
        "from coqui_stt_training.util.downloader import maybe_download\n",
        "\n",
        "def download_pretrained_model():\n",
        "    model_dir=\"english/\"\n",
        "    if not os.path.exists(\"english/coqui-stt-1.1.0-checkpoint\"):\n",
        "        maybe_download(\"model.tar.gz\", model_dir, \"https://github.com/coqui-ai/STT/releases/download/v1.1.0/coqui-stt-1.1.0-checkpoint.tar.gz\")\n",
        "        print('\\nNo extracted pre-trained model found. Extracting now...')\n",
        "        tar = tarfile.open(\"english/model.tar.gz\")\n",
        "        tar.extractall(\"english/\")\n",
        "        tar.close()\n",
        "    else:\n",
        "        print('Found pre-trained 🐸STT model, skipping download.')\n",
        "\n",
        "def download_language_model():\n",
        "    model_dir=\"english/\"\n",
        "    if not os.path.exists(\"english/huge-vocabulary.scorer\"):\n",
        "        maybe_download(\"huge-vocabulary.scorer\", model_dir, \"https://github.com/coqui-ai/STT-models/releases/download/english/coqui/v1.0.0-huge-vocab/huge-vocabulary.scorer\")\n",
        "    else:\n",
        "        print('Found 🐸STT language model, skipping download.')\n",
        "\n",
        "# Download + extract pre-trained English model\n",
        "download_pretrained_model()\n",
        "download_language_model()"
      ],
      "id": "608d203f"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ae82fd75"
      },
      "source": [
        "## ✅ Customize to your Voice\n",
        "\n",
        "Let's kick off a training run 🚀🚀🚀 and fine-tune the pre-trained 🐸STT model to your unique voice."
      ],
      "id": "ae82fd75"
    },
    {
      "cell_type": "code",
      "source": [
        "from coqui_stt_training.util.config import initialize_globals_from_args\n",
        "from coqui_stt_training.train import train\n",
        "import glob\n",
        "\n",
        "TRAIN_CSV = glob.glob(\"/content/takeout_*/train.csv\")\n",
        "DEV_CSV = glob.glob(\"/content/takeout_*/dev.csv\")\n",
        "\n",
        "initialize_globals_from_args(\n",
        "    load_checkpoint_dir=\"english/coqui-stt-1.1.0-checkpoint\",\n",
        "    save_checkpoint_dir=\"my-model/checkpoints\",\n",
        "    alphabet_config_path=\"english/coqui-stt-1.1.0-checkpoint/alphabet.txt\",\n",
        "    train_files=TRAIN_CSV,\n",
        "    dev_files=DEV_CSV,\n",
        "    epochs=10,\n",
        "    load_cudnn=True,\n",
        "    train_batch_size=32,\n",
        "    dev_batch_size=32,\n",
        ")\n",
        "\n",
        "train()"
      ],
      "metadata": {
        "id": "ZG01HHCpi_6S"
      },
      "id": "ZG01HHCpi_6S",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9f6dc959"
      },
      "source": [
        "## ✅ Test the model\n",
        "\n",
        "We made it! 🙌\n",
        "\n",
        "Let's kick off the testing run, and see how the model performs.\n",
        "\n",
        "First, let's see how well the off-the-shelf English model recognizes your voice."
      ],
      "id": "9f6dc959"
    },
    {
      "cell_type": "code",
      "source": [
        "# Test STT v1.0 on your voice data\n",
        "\n",
        "from coqui_stt_training.util.config import initialize_globals_from_args\n",
        "from coqui_stt_training.evaluate import test\n",
        "import glob\n",
        "\n",
        "TEST_CSV = glob.glob(\"/content/takeout_*/test.csv\")\n",
        "\n",
        "initialize_globals_from_args(\n",
        "    load_checkpoint_dir=\"english/coqui-stt-1.1.0-checkpoint\",\n",
        "    scorer_path=\"english/huge-vocabulary.scorer\",\n",
        "    test_files = TEST_CSV,\n",
        "    test_batch_size = 32,\n",
        ")\n",
        "\n",
        "test()"
      ],
      "metadata": {
        "id": "o9prwww3upwO"
      },
      "id": "o9prwww3upwO",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, let's see how much better ✨ your new custom model ✨ performs 🤩"
      ],
      "metadata": {
        "id": "txez_adC1b_m"
      },
      "id": "txez_adC1b_m"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dd42bc7a"
      },
      "source": [
        "# Test your custom model on your held-out test data\n",
        "\n",
        "from coqui_stt_training.util.config import initialize_globals_from_args\n",
        "from coqui_stt_training.evaluate import test\n",
        "import glob\n",
        "\n",
        "TEST_CSV = glob.glob(\"/content/takeout_*/test.csv\")\n",
        "\n",
        "initialize_globals_from_args(\n",
        "    load_checkpoint_dir=\"my-model/checkpoints\",\n",
        "    scorer_path=\"english/huge-vocabulary.scorer\",\n",
        "    test_files = TEST_CSV,\n",
        "    test_batch_size=32,\n",
        ")\n",
        "\n",
        "test()"
      ],
      "id": "dd42bc7a",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 🤷‍♀️ What next?\n",
        "\n",
        "Now that you've trained your own personal Speech-to-Text model, you can download the `my-model/checkpoints` directory to your local computer, and use it on your own computer without an internet connection!\n",
        "\n",
        "The sky's the limit for what's possible, but if you want some inspiration, check out the [STT-examples](https://github.com/coqui-ai/stt-examples) collection of community examples with STT.\n",
        "\n",
        "Happy hacking!"
      ],
      "metadata": {
        "id": "JMUdCaej647C"
      },
      "id": "JMUdCaej647C"
    },
    {
      "cell_type": "markdown",
      "source": [
        "### from 🐸 with 💚"
      ],
      "metadata": {
        "id": "dtB8Ex1A9BS3"
      },
      "id": "dtB8Ex1A9BS3"
    }
  ]
}
