{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "distilGPT2-finetuned-MA-Meditations.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyMcK4WOlV6qfNTWu+JhSj+T",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrm8488/shared_colab_notebooks/blob/master/distilGPT2_finetuned_MA_Meditations.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "niZE8AHRdiEx",
        "colab_type": "text"
      },
      "source": [
        "# Fine tune distilGPT-2 on Marco Aurelio Medidations for text generation\n",
        "\n",
        "> Author: [Manuel Romero / mrm8488](https://twitter.com/mrm8488)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d-YqwhZzVZaU",
        "colab_type": "text"
      },
      "source": [
        "## Get the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hOss2aihVa_3",
        "colab_type": "code",
        "outputId": "c32083d7-3281-420f-8696-5650d375fffa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "!wget https://raw.githubusercontent.com/GITenberg/Meditations_2680/master/2680.txt -O text.txt"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-05-20 14:22:29--  https://raw.githubusercontent.com/GITenberg/Meditations_2680/master/2680.txt\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 425173 (415K) [text/plain]\n",
            "Saving to: ‘text.txt’\n",
            "\n",
            "\rtext.txt              0%[                    ]       0  --.-KB/s               \rtext.txt            100%[===================>] 415.21K  --.-KB/s    in 0.06s   \n",
            "\n",
            "2020-05-20 14:22:29 (6.72 MB/s) - ‘text.txt’ saved [425173/425173]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZndJ6XHhqRw",
        "colab_type": "text"
      },
      "source": [
        "### Remove book index, introduction an so on"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6vFqJ4sUV7kG",
        "colab_type": "code",
        "outputId": "2139e67d-ed35-4785-a53e-b7c566e04929",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!wc -l text.txt"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "7212 text.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ptp9JF77XL72",
        "colab_type": "code",
        "outputId": "a5571076-69fd-4b6a-a877-48ba67ff459c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!head -88 text.txt"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The Project Gutenberg EBook of Meditations, by Marcus Aurelius\r\n",
            "\r\n",
            "This eBook is for the use of anyone anywhere at no cost and with\r\n",
            "almost no restrictions whatsoever.  You may copy it, give it away or\r\n",
            "re-use it under the terms of the Project Gutenberg License included\r\n",
            "with this eBook or online at www.gutenberg.org\r\n",
            "\r\n",
            "\r\n",
            "Title: Meditations\r\n",
            "\r\n",
            "Author: Marcus Aurelius\r\n",
            "\r\n",
            "Posting Date: December 25, 2008 [EBook #2680]\r\n",
            "Release Date: June, 2001\r\n",
            "\r\n",
            "Language: English\r\n",
            "\r\n",
            "Character set encoding: ASCII\r\n",
            "\r\n",
            "*** START OF THIS PROJECT GUTENBERG EBOOK MEDITATIONS ***\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "Produced by J. Boulton\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "MEDITATIONS\r\n",
            "\r\n",
            "By Marcus Aurelius\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "CONTENTS\r\n",
            "\r\n",
            "\r\n",
            "     NOTES\r\n",
            "\r\n",
            "     INTRODUCTION\r\n",
            "\r\n",
            "     FIRST BOOK\r\n",
            "\r\n",
            "     SECOND BOOK\r\n",
            "\r\n",
            "     THIRD BOOK\r\n",
            "\r\n",
            "     FOURTH BOOK\r\n",
            "\r\n",
            "     FIFTH BOOK\r\n",
            "\r\n",
            "     SIXTH BOOK\r\n",
            "\r\n",
            "     SEVENTH BOOK\r\n",
            "\r\n",
            "     EIGHTH BOOK\r\n",
            "\r\n",
            "     NINTH BOOK\r\n",
            "\r\n",
            "     TENTH BOOK\r\n",
            "\r\n",
            "     ELEVENTH BOOK\r\n",
            "\r\n",
            "     TWELFTH BOOK\r\n",
            "\r\n",
            "     APPENDIX\r\n",
            "\r\n",
            "     GLOSSARY\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "Original Transcriber's Notes:\r\n",
            "\r\n",
            "This text was scanned by J. Boulton using Textbridge OCR. The Greek\r\n",
            "portions of the text have been added by hand and they will require the\r\n",
            "standard \"Symbol\" font \"symbol.ttf\" to be installed in the system fonts\r\n",
            "folder. This is a standard Windows font, so should be present on most\r\n",
            "systems. To contact the scanner e-mail: magicjon@ic24.net INTRODUCTION\r\n",
            "This is the Plain Text version, see medma10h.txt or .zip for the HTML\r\n",
            "version with the various symbols mentioned above.\r\n",
            "\r\n",
            "\r\n",
            "\r\n",
            "\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vlgvv5boXpNZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cat text.txt | tail -7124 >> train.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AEWjk9gqXQ31",
        "colab_type": "code",
        "outputId": "3979001a-1961-4e70-def2-b4135863d9ce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        }
      },
      "source": [
        "!head -20 train.txt"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INTRODUCTION\r\n",
            "\r\n",
            "\r\n",
            "MARCUS AURELIUS ANTONINUS was born on April 26, A.D. 121. His real name\r\n",
            "was M. Annius Verus, and he was sprung of a noble family which claimed\r\n",
            "descent from Numa, second King of Rome. Thus the most religious of\r\n",
            "emperors came of the blood of the most pious of early kings. His father,\r\n",
            "Annius Verus, had held high office in Rome, and his grandfather, of\r\n",
            "the same name, had been thrice Consul. Both his parents died young, but\r\n",
            "Marcus held them in loving remembrance. On his father's death Marcus\r\n",
            "was adopted by his grandfather, the consular Annius Verus, and there was\r\n",
            "deep love between these two. On the very first page of his book Marcus\r\n",
            "gratefully declares how of his grandfather he had learned to be gentle\r\n",
            "and meek, and to refrain from all anger and passion. The Emperor Hadrian\r\n",
            "divined the fine character of the lad, whom he used to call not Verus\r\n",
            "but Verissimus, more Truthful than his own name. He advanced Marcus to\r\n",
            "equestrian rank when six years of age, and at the age of eight made him\r\n",
            "a member of the ancient Salian priesthood. The boy's aunt, Annia Galeria\r\n",
            "Faustina, was married to Antoninus Pius, afterwards emperor. Hence it\r\n",
            "came about that Antoninus, having no son, adopted Marcus, changing his\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cIIMcSigX3kC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!rm -rf text.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aTifIFDQX-fH",
        "colab_type": "text"
      },
      "source": [
        "## Install HF Transformers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A95hoiPoX79s",
        "colab_type": "code",
        "outputId": "d081cb1d-3456-4d20-f7b3-d2b498610b9b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "!git clone https://github.com/huggingface/transformers.git"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'transformers'...\n",
            "remote: Enumerating objects: 19, done.\u001b[K\n",
            "remote: Counting objects: 100% (19/19), done.\u001b[K\n",
            "remote: Compressing objects: 100% (13/13), done.\u001b[K\n",
            "remote: Total 26714 (delta 9), reused 5 (delta 5), pack-reused 26695\u001b[K\n",
            "Receiving objects: 100% (26714/26714), 25.01 MiB | 31.04 MiB/s, done.\n",
            "Resolving deltas: 100% (18596/18596), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pHxOa9DLYLeU",
        "colab_type": "code",
        "outputId": "54bf3d29-bfc0-4855-b237-bc6bc520ee21",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "!pip install -q ./transformers"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K     |████████████████████████████████| 3.8MB 3.4MB/s \n",
            "\u001b[K     |████████████████████████████████| 1.1MB 42.3MB/s \n",
            "\u001b[K     |████████████████████████████████| 890kB 32.0MB/s \n",
            "\u001b[?25h  Building wheel for transformers (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5zpu389Ih0Kb",
        "colab_type": "text"
      },
      "source": [
        "## Fine tuning the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IWc0gU2kYb5i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python /content/transformers/examples/language-modeling/run_language_modeling.py \\\n",
        "--model_type=gpt2 \\\n",
        "--model_name_or_path=distilgpt2 \\\n",
        "--do_train \\\n",
        "--train_data_file=/content/train.txt \\\n",
        "--num_train_epochs 100 \\\n",
        "--output_dir model_output \\\n",
        "--overwrite_output_dir \\\n",
        "--save_steps 20000 \\\n",
        "--per_gpu_train_batch_size 4\n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n2Ui_GI2h-yF",
        "colab_type": "text"
      },
      "source": [
        "## Generate text with our fine tuned model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nv7UrthKjVUw",
        "colab_type": "code",
        "outputId": "5803a9d8-2e77-4113-c721-7a33b14a6e23",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!python /content/transformers/examples/text-generation/run_generation.py \\\n",
        "--model_type=gpt2 \\\n",
        "--model_name_or_path=/content/model_output \\\n",
        "--num_return_sequences 5 \\\n",
        "--length 100 \\\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-05-20 15:06:05.250849: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "05/20/2020 15:06:06 - INFO - transformers.tokenization_utils -   Model name '/content/model_output' not found in model shortcut name list (gpt2, gpt2-medium, gpt2-large, gpt2-xl, distilgpt2). Assuming '/content/model_output' is a path, a model identifier, or url to a directory containing tokenizer files.\n",
            "05/20/2020 15:06:06 - INFO - transformers.tokenization_utils -   Didn't find file /content/model_output/added_tokens.json. We won't load it.\n",
            "05/20/2020 15:06:06 - INFO - transformers.tokenization_utils -   loading file /content/model_output/vocab.json\n",
            "05/20/2020 15:06:06 - INFO - transformers.tokenization_utils -   loading file /content/model_output/merges.txt\n",
            "05/20/2020 15:06:06 - INFO - transformers.tokenization_utils -   loading file None\n",
            "05/20/2020 15:06:06 - INFO - transformers.tokenization_utils -   loading file /content/model_output/special_tokens_map.json\n",
            "05/20/2020 15:06:06 - INFO - transformers.tokenization_utils -   loading file /content/model_output/tokenizer_config.json\n",
            "05/20/2020 15:06:06 - INFO - transformers.configuration_utils -   loading configuration file /content/model_output/config.json\n",
            "05/20/2020 15:06:06 - INFO - transformers.configuration_utils -   Model config GPT2Config {\n",
            "  \"_num_labels\": 1,\n",
            "  \"activation_function\": \"gelu_new\",\n",
            "  \"architectures\": [\n",
            "    \"GPT2LMHeadModel\"\n",
            "  ],\n",
            "  \"attn_pdrop\": 0.1,\n",
            "  \"bos_token_id\": 50256,\n",
            "  \"embd_pdrop\": 0.1,\n",
            "  \"eos_token_id\": 50256,\n",
            "  \"id2label\": {\n",
            "    \"0\": \"LABEL_0\"\n",
            "  },\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"label2id\": {\n",
            "    \"LABEL_0\": 0\n",
            "  },\n",
            "  \"layer_norm_epsilon\": 1e-05,\n",
            "  \"model_type\": \"gpt2\",\n",
            "  \"n_ctx\": 1024,\n",
            "  \"n_embd\": 768,\n",
            "  \"n_head\": 12,\n",
            "  \"n_layer\": 6,\n",
            "  \"n_positions\": 1024,\n",
            "  \"resid_pdrop\": 0.1,\n",
            "  \"summary_activation\": null,\n",
            "  \"summary_first_dropout\": 0.1,\n",
            "  \"summary_proj_to_labels\": true,\n",
            "  \"summary_type\": \"cls_index\",\n",
            "  \"summary_use_proj\": true,\n",
            "  \"task_specific_params\": {\n",
            "    \"text-generation\": {\n",
            "      \"do_sample\": true,\n",
            "      \"max_length\": 50\n",
            "    }\n",
            "  },\n",
            "  \"vocab_size\": 50257\n",
            "}\n",
            "\n",
            "05/20/2020 15:06:06 - INFO - transformers.modeling_utils -   loading weights file /content/model_output/pytorch_model.bin\n",
            "05/20/2020 15:06:13 - INFO - __main__ -   Namespace(device=device(type='cuda'), k=0, length=100, model_name_or_path='/content/model_output', model_type='gpt2', n_gpu=1, no_cuda=False, num_return_sequences=5, p=0.9, padding_text='', prompt='', repetition_penalty=1.0, seed=42, stop_token=None, temperature=1.0, xlm_language='')\n",
            "Model prompt >>> The meaning of the life is\n",
            "05/20/2020 15:06:33 - WARNING - transformers.modeling_utils -   Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence\n",
            "=== GENERATED SEQUENCE 1 ===\n",
            "The meaning of the life is not obscure; but the word is certainly understood to mean that\n",
            "man in general was made for man to do better than he was. Of all the virtues that\n",
            "are properly distinguished from that which is now present, there are two\n",
            "that stand in dispute. First, that life is not easy, if many years of it be\n",
            "that is, will soon be out of sight. And secondly, that life is not as easy for ordinary\n",
            "workers as he is a millstone be. For even\n",
            "=== GENERATED SEQUENCE 2 ===\n",
            "The meaning of the life is,\n",
            "\"I conceive it not so,\" he continued. \"But, then, when death cut short my\n",
            "life, I prayed the elements would come together again, and bring back\n",
            "new Providence.\n",
            "\n",
            "XXXI. If you have any memories of that day, remember it so shall\n",
            "remain fresh and clear for all who are long since dead, and all who know\n",
            "how long the world stands in need of an explanation, may be able to comfort\n",
            "all. But remember,\n",
            "=== GENERATED SEQUENCE 3 ===\n",
            "The meaning of the life is: 'For,\n",
            "thou art not able to live long.' 'Foolishness, slander,\n",
            "etc.' 'Aman,' says Plato,' 'thou must have an opinion of\n",
            "those things which we ought to be sorry for.' 'But for\n",
            "them, as long as we live, it will be our hurt to grant any benefit.' 'Or as\n",
            "a lion, if he shall only be handsome, the colour of his skin, if he shall be\n",
            "white.'\n",
            "=== GENERATED SEQUENCE 4 ===\n",
            "The meaning of the life is, 'to take away that from thee\n",
            "that very soul.' 'It is not about power; it is about courage.'\n",
            "\n",
            "So it runs, as if the winds beat in favour of the strong, the wind and\n",
            "the rain beat us down. Then again, as in the story of Euripides, the\n",
            "Telus, the writer of Aristophanes, who is rightly accused, writes:(1)\n",
            "'I see that my suspicion is, that it is not about fear\n",
            "=== GENERATED SEQUENCE 5 ===\n",
            "The meaning of the life is, 'To\n",
            "thou mayest be able to rectify, or to ease thy mind.' If thou dost 'rigour thyself, or\n",
            "thou amend, thou mayest find comfort in thy desire.' But it is a\n",
            "thing else, that thy heart may always be set upon. 'Thou\n",
            "must speed, if thou shalt ease thy way, if thou shalt ease thy way,\n",
            "if thou shalt not fear, when thou doest ease thy way.' As for\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JspKNYq4pn4x",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}