{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "BETO_getting_started.ipynb",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrm8488/shared_colab_notebooks/blob/master/BETO_getting_started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p2LNuj2a59BL",
        "colab_type": "text"
      },
      "source": [
        "#Getting started with BETO\n",
        "> Colab by [Manuel Romero](https://twitter.com/mrm8488)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SkesDoqe6Dku",
        "colab_type": "text"
      },
      "source": [
        "# BETO: Spanish BERT\n",
        "\n",
        "BETO is a [BERT model](https://github.com/google-research/bert) trained on a [big Spanish corpus](https://github.com/josecannete/spanish-corpora). BETO is of size similar to a BERT-Base and was trained with the Whole Word Masking technique. Below you find Tensorflow and Pytorch checkpoints for the uncased and cased versions, as well as some results for Spanish benchmarks comparing BETO with [Multilingual BERT](https://github.com/google-research/bert/blob/master/multilingual.md) as well as other (not BERT-based) models."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "For8_AVDd3rc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 654
        },
        "outputId": "732195c9-7f68-42f5-8916-c060a7a020f0"
      },
      "source": [
        "!wget https://users.dcc.uchile.cl/~jperez/beto/cased_2M/pytorch_weights.tar.gz \n",
        "!wget https://users.dcc.uchile.cl/~jperez/beto/cased_2M/vocab.txt \n",
        "!wget https://users.dcc.uchile.cl/~jperez/beto/cased_2M/config.json \n",
        "!tar -xzvf pytorch_weights.tar.gz\n",
        "!mv config.json pytorch/.\n",
        "!mv vocab.txt pytorch/."
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-12-30 20:20:39--  https://users.dcc.uchile.cl/~jperez/beto/cased_2M/pytorch_weights.tar.gz\n",
            "Resolving users.dcc.uchile.cl (users.dcc.uchile.cl)... 200.9.99.211, 192.80.24.4\n",
            "Connecting to users.dcc.uchile.cl (users.dcc.uchile.cl)|200.9.99.211|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 409871727 (391M) [application/x-gzip]\n",
            "Saving to: ‘pytorch_weights.tar.gz’\n",
            "\n",
            "pytorch_weights.tar 100%[===================>] 390.88M  3.38MB/s    in 2m 27s  \n",
            "\n",
            "2019-12-30 20:23:07 (2.66 MB/s) - ‘pytorch_weights.tar.gz’ saved [409871727/409871727]\n",
            "\n",
            "--2019-12-30 20:23:09--  https://users.dcc.uchile.cl/~jperez/beto/cased_2M/vocab.txt\n",
            "Resolving users.dcc.uchile.cl (users.dcc.uchile.cl)... 200.9.99.211, 192.80.24.4\n",
            "Connecting to users.dcc.uchile.cl (users.dcc.uchile.cl)|200.9.99.211|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 242349 (237K) [text/plain]\n",
            "Saving to: ‘vocab.txt’\n",
            "\n",
            "vocab.txt           100%[===================>] 236.67K   289KB/s    in 0.8s    \n",
            "\n",
            "2019-12-30 20:23:11 (289 KB/s) - ‘vocab.txt’ saved [242349/242349]\n",
            "\n",
            "--2019-12-30 20:23:13--  https://users.dcc.uchile.cl/~jperez/beto/cased_2M/config.json\n",
            "Resolving users.dcc.uchile.cl (users.dcc.uchile.cl)... 200.9.99.211, 192.80.24.4\n",
            "Connecting to users.dcc.uchile.cl (users.dcc.uchile.cl)|200.9.99.211|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 313 [application/json]\n",
            "Saving to: ‘config.json’\n",
            "\n",
            "config.json         100%[===================>]     313  --.-KB/s    in 0s      \n",
            "\n",
            "2019-12-30 20:23:14 (55.0 MB/s) - ‘config.json’ saved [313/313]\n",
            "\n",
            "pytorch/\n",
            "pytorch/pytorch_model.bin\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1q1CU0O06ELS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install transformers > /dev/null"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LWy074dJ6HW3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Avka9_xF6Ja2",
        "colab_type": "code",
        "outputId": "d3776980-5fe2-4f20-a2bf-c664cb923406",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 65
        }
      },
      "source": [
        "from transformers import BertTokenizer, BertForMaskedLM"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bwS9Zu8b6Nd8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tokenizer = BertTokenizer.from_pretrained(\"pytorch/\", do_lower_case=False)\n",
        "model = BertForMaskedLM.from_pretrained(\"pytorch/\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w3nzzyjG6OUA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.eval()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eTmaZgv06STH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "text = \"[CLS] Para solucionar los [MASK] de Chile, el presidente debe [MASK] de inmediato. [SEP]\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZqsADzQI6UcH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "masked_indxs = (4,11)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rvWXuR6b6Vft",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "indexed_tokens = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text))\n",
        "tokens_tensor = torch.tensor([indexed_tokens])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M9eUOzvE6XWq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "predictions = model(tokens_tensor)[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tUvd9IxG6aoz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "55953024-2c4c-4303-c8f4-0180e1fb6eb2"
      },
      "source": [
        "for i, midx in enumerate(masked_indxs):\n",
        "  idxs = torch.argsort(predictions[0,midx], descending=True)\n",
        "  predicted_token = tokenizer.convert_ids_to_tokens(idxs[:5])\n",
        "  print('MASKS', i, ':', predicted_token)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "MASKS 0 : ['problemas', 'conflictos', 'asuntos', 'males', 'temas']\n",
            "MASKS 1 : ['renunciar', 'actuar', 'intervenir', 'regresar', 'asumir']\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}