{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Sahel-Eskandar/DataScience-Projects/blob/main/16_TextClassificationSentimentalAnalysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ziJQAjH1bTzf"
      },
      "source": [
        "# Introduction\n",
        "\n",
        "Text classification and sentiment analysis are two related techniques in natural language processing that involve automatically analyzing and categorizing text data based on its content.\n",
        "\n",
        "Text classification, also known as text categorization, is the process of assigning a predefined category or label to a given text document. The goal of text classification is to automatically classify large volumes of text data based on their content, making it easier to organize, search, and analyze. For example, a news article might be classified into categories such as politics, sports, entertainment, etc.\n",
        "\n",
        "Sentiment analysis, also known as opinion mining, is a type of text classification that focuses specifically on identifying the emotional tone of a given piece of text. The goal of sentiment analysis is to automatically determine whether a given text expresses a positive, negative, or neutral sentiment. For example, a customer review of a product might be classified as positive, negative, or neutral based on the language used to describe the product.\n",
        "\n",
        "Both text classification and sentiment analysis are important tools for businesses, organizations, and researchers who want to make sense of large volumes of text data. They can be used to identify trends, monitor social media sentiment, analyze customer feedback, and much more.\n",
        "\n",
        "Text classification and sentiment analysis involve a combination of NLP and data analysis techniques to analyze and categorize large volumes of text data. Following techniques are commonly used:\n",
        "\n",
        "1. Text preprocessing: This involves cleaning and transforming the raw text data to prepare it for analysis. Techniques such as tokenization (breaking text into words or phrases), stopword removal (removing common words that don't carry much meaning), and stemming (reducing words to their base form) can be used to simplify the text data.\n",
        "\n",
        "2. Feature extraction: This involves selecting the most important features (words or phrases) from the preprocessed text data that are likely to be relevant for classification. Techniques such as bag-of-words (counting the frequency of each word in a document), TF-IDF (weighing words by their importance in a document), and word embeddings (representing words as vectors in a high-dimensional space) can be used to extract features.\n",
        "\n",
        "3. Machine learning algorithms: This involves training a machine learning model on the preprocessed and feature-extracted text data to predict the category or sentiment of new text data. Common machine learning algorithms used for text classification and sentiment analysis include Naive Bayes, Support Vector Machines (SVM), Random Forests, and Deep Learning models like Recurrent Neural Networks (RNN) and Convolutional Neural Networks (CNN).This notebook will focus on text-classification and sentiment analysis. We will go through all major NLP and dat analysis techniques, some of which include:\n",
        "    - LSTMs\n",
        "    - Transformers (such as BERT)\n",
        "    - Naive Bayes\n",
        "    - XGBoost \n",
        "\n",
        "4. Evaluation metrics: This involves measuring the performance of the text classification and sentiment analysis models on a test dataset. Metrics such as accuracy, precision, recall, F1 score, and confusion matrix can be used to evaluate the performance of the models.\n",
        "\n",
        "5. Model optimization: This involves fine-tuning the machine learning models to improve their performance on the test dataset. Techniques such as hyperparameter tuning (adjusting model parameters such as learning rate, regularization, etc.), feature selection (selecting the most important features for the model), and ensemble learning (combining multiple models to improve performance) can be used to optimize the models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oPWzrwbYbVky"
      },
      "source": [
        "Data Source: \n",
        "\n",
        "kaggle kernels output namansood/nlp-xgboost-bert-glove-rnns-vectorizers -p /path/to/dest\n",
        "\n",
        "https://www.kaggle.com/code/namansood/nlp-xgboost-bert-glove-rnns-vectorizers/data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "id": "2VK8ZjqHbTzj",
        "tags": []
      },
      "outputs": [],
      "source": [
        "import numpy as np \n",
        "import pandas as pd \n",
        "\n",
        "\n",
        "import os\n",
        "for dirname, _, filenames in os.walk('/kaggle/input'):\n",
        "    for filename in filenames:\n",
        "        print(os.path.join(dirname, filename))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "tGHolVNhbTzk",
        "tags": []
      },
      "outputs": [],
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "encoding_errors='ignore'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eD253UZqUrZm",
        "outputId": "c84a43be-834a-4a97-b3e0-de090ca9b455"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: wordcloud in /usr/local/lib/python3.8/dist-packages (1.8.2.2)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from wordcloud) (3.5.3)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.8/dist-packages (from wordcloud) (8.4.0)\n",
            "Requirement already satisfied: numpy>=1.6.1 in /usr/local/lib/python3.8/dist-packages (from wordcloud) (1.22.4)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.8/dist-packages (from matplotlib->wordcloud) (2.8.2)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->wordcloud) (4.38.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->wordcloud) (1.4.4)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->wordcloud) (0.11.0)\n",
            "Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->wordcloud) (3.0.9)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->wordcloud) (23.0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil>=2.7->matplotlib->wordcloud) (1.15.0)\n"
          ]
        }
      ],
      "source": [
        "!pip install wordcloud"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DcBr7GN6UrZn",
        "outputId": "e8466f7e-5eeb-433b-c732-8d25e67edde2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.8/dist-packages (3.7)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.8/dist-packages (from nltk) (1.2.0)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.8/dist-packages (from nltk) (2022.6.2)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.8/dist-packages (from nltk) (8.1.3)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from nltk) (4.64.1)\n"
          ]
        }
      ],
      "source": [
        "!pip install nltk"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "okKEp6bZUrZn",
        "outputId": "1d90c523-db55-497f-cc97-5947a217e178"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: spacy in /usr/local/lib/python3.8/dist-packages (3.4.4)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.8/dist-packages (from spacy) (2.0.8)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in /usr/local/lib/python3.8/dist-packages (from spacy) (0.10.1)\n",
            "Requirement already satisfied: typer<0.8.0,>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from spacy) (0.7.0)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy) (3.0.8)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.8/dist-packages (from spacy) (1.22.4)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.8/dist-packages (from spacy) (1.0.9)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.8/dist-packages (from spacy) (0.10.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from spacy) (57.4.0)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy) (2.0.7)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from spacy) (1.0.4)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.10 in /usr/local/lib/python3.8/dist-packages (from spacy) (3.0.12)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /usr/local/lib/python3.8/dist-packages (from spacy) (1.10.5)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from spacy) (23.0)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.8/dist-packages (from spacy) (3.3.0)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /usr/local/lib/python3.8/dist-packages (from spacy) (8.1.7)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.8/dist-packages (from spacy) (6.3.0)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from spacy) (3.1.2)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.8/dist-packages (from spacy) (2.4.6)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.8/dist-packages (from spacy) (4.64.1)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.8/dist-packages (from spacy) (2.25.1)\n",
            "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.8/dist-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy) (4.5.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (1.26.14)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2.10)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (4.0.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2022.12.7)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy) (0.7.9)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy) (0.0.4)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.8/dist-packages (from typer<0.8.0,>=0.3.0->spacy) (8.1.3)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->spacy) (2.1.2)\n"
          ]
        }
      ],
      "source": [
        "!pip install spacy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m6k799ROUrZn",
        "outputId": "badf5967-2c3e-4cdc-d21a-8537d85df6c2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: keras in /usr/local/lib/python3.8/dist-packages (2.11.0)\n"
          ]
        }
      ],
      "source": [
        "!pip install keras"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qbqzE-krUrZn",
        "outputId": "6810110c-bc36-4dea-9516-1e7786b63ed7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: tensorflow in /usr/local/lib/python3.8/dist-packages (2.11.0)\n",
            "Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.15.0)\n",
            "Requirement already satisfied: gast<=0.4.0,>=0.2.1 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (0.4.0)\n",
            "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (3.1.0)\n",
            "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (0.31.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (4.5.0)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.15.0)\n",
            "Requirement already satisfied: protobuf<3.20,>=3.9.2 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (3.19.6)\n",
            "Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.22.4)\n",
            "Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.4.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (3.3.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from tensorflow) (23.0)\n",
            "Requirement already satisfied: keras<2.12,>=2.11.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (2.11.0)\n",
            "Requirement already satisfied: tensorflow-estimator<2.12,>=2.11.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (2.11.0)\n",
            "Requirement already satisfied: flatbuffers>=2.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (23.1.21)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from tensorflow) (57.4.0)\n",
            "Requirement already satisfied: tensorboard<2.12,>=2.11 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (2.11.2)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (2.2.0)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.51.3)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (0.2.0)\n",
            "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.6.3)\n",
            "Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (15.0.6.1)\n",
            "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.8/dist-packages (from astunparse>=1.6.0->tensorflow) (0.38.4)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (3.4.1)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (2.2.3)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (0.6.1)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (0.4.6)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (1.8.1)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (2.16.1)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (2.25.1)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow) (4.9)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow) (5.3.0)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow) (0.2.8)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.12,>=2.11->tensorflow) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.8/dist-packages (from markdown>=2.6.8->tensorboard<2.12,>=2.11->tensorflow) (6.0.0)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow) (4.0.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow) (1.26.14)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow) (2.10)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.8/dist-packages (from werkzeug>=1.0.1->tensorboard<2.12,>=2.11->tensorflow) (2.1.2)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.12,>=2.11->tensorflow) (3.15.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.8/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow) (0.4.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.12,>=2.11->tensorflow) (3.2.2)\n"
          ]
        }
      ],
      "source": [
        "!pip install tensorflow"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "ZYZQCUolbTzl",
        "tags": []
      },
      "outputs": [],
      "source": [
        "import re\n",
        "import string\n",
        "import numpy as np \n",
        "import random\n",
        "import pandas as pd \n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "%matplotlib inline\n",
        "from plotly import graph_objs as go\n",
        "import plotly.express as px\n",
        "import plotly.figure_factory as ff\n",
        "from collections import Counter\n",
        "\n",
        "from PIL import Image\n",
        "from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator\n",
        "\n",
        "\n",
        "import nltk\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "from tqdm import tqdm\n",
        "import os\n",
        "import nltk\n",
        "import spacy\n",
        "import random\n",
        "from spacy.util import compounding\n",
        "from spacy.util import minibatch\n",
        "\n",
        "from collections import defaultdict\n",
        "from collections import Counter\n",
        "\n",
        "from sklearn.metrics import precision_score, recall_score, f1_score, classification_report,accuracy_score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x-d9ZGOBUrZn",
        "outputId": "26589425-4b7b-488f-c5f5-b4db7cdaa4fe"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting Keras-Preprocessing\n",
            "  Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.6/42.6 KB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.8/dist-packages (from Keras-Preprocessing) (1.15.0)\n",
            "Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.8/dist-packages (from Keras-Preprocessing) (1.22.4)\n",
            "Installing collected packages: Keras-Preprocessing\n",
            "Successfully installed Keras-Preprocessing-1.1.2\n"
          ]
        }
      ],
      "source": [
        "!pip install Keras-Preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "CQG1qs7wf-8r",
        "tags": []
      },
      "outputs": [],
      "source": [
        "import keras\n",
        "from keras.models import Sequential\n",
        "from keras.initializers import Constant\n",
        "from keras.layers import LSTM, Embedding,BatchNormalization, Dense, TimeDistributed, Dropout, Bidirectional, Flatten, GlobalMaxPool1D\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras_preprocessing.sequence import pad_sequences\n",
        "#from keras.layers.embeddings import Embedding\n",
        "from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau\n",
        "from keras.optimizers import Adam"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "vui-pD2lbTzl",
        "tags": []
      },
      "outputs": [],
      "source": [
        "# Defining the global variables for the color schemes we will incorporate\n",
        "pblue = \"#496595\"\n",
        "pb2 = \"#85a1c1\"\n",
        "pb3 = \"#3f4d63\"\n",
        "pg = \"#c6ccd8\"\n",
        "pb = \"#202022\"\n",
        "pbg = \"#f4f0ea\"\n",
        "\n",
        "pgreen = px.colors.qualitative.Plotly[2]"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "url = 'https://raw.githubusercontent.com/Sahel-Eskandar/DataScience-Projects/main/Data/spam.csv'\n",
        "df = pd.read_csv(url, encoding = \"ISO-8859-1\", error_bad_lines=False)"
      ],
      "metadata": {
        "id": "ajp12J6Lblgx"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "oUG06LinbTzm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "67bfab14-b867-4f93-8ca3-e56b5989e35c"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "v1               0\n",
              "v2               0\n",
              "Unnamed: 2    5522\n",
              "Unnamed: 3    5560\n",
              "Unnamed: 4    5566\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ],
      "source": [
        "df.isnull().sum()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "5b0xb9XXbTzm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4a638479-1d4a-4c33-a86e-a80504d76a2b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "v1            object\n",
              "v2            object\n",
              "Unnamed: 2    object\n",
              "Unnamed: 3    object\n",
              "Unnamed: 4    object\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ],
      "source": [
        "df.dtypes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "hSrHWTdkbTzn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "47f045ea-3a3b-406d-e691-033fd67cc9d6"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "v1               0\n",
              "v2               0\n",
              "Unnamed: 2    5522\n",
              "Unnamed: 3    5560\n",
              "Unnamed: 4    5566\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "df.isnull().sum()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "7FpcK8PIbTzn",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "c387cbe0-aacd-40bd-deb1-8fe9064b6997"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     v1                                                 v2\n",
              "0   ham  Go until jurong point, crazy.. Available only ...\n",
              "1   ham                      Ok lar... Joking wif u oni...\n",
              "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
              "3   ham  U dun say so early hor... U c already then say...\n",
              "4   ham  Nah I don't think he goes to usf, he lives aro..."
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f1174e21-dc3b-4ce0-bc01-c6d1ea2cc994\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>v1</th>\n",
              "      <th>v2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f1174e21-dc3b-4ce0-bc01-c6d1ea2cc994')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f1174e21-dc3b-4ce0-bc01-c6d1ea2cc994 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f1174e21-dc3b-4ce0-bc01-c6d1ea2cc994');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ],
      "source": [
        "df.dropna(axis=1, inplace=True)\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "79sOaWlWbTzn",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "72bb6d63-3fb8-447d-fd24-27b26f477829"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  label                                               text\n",
              "0   ham  Go until jurong point, crazy.. Available only ...\n",
              "1   ham                      Ok lar... Joking wif u oni...\n",
              "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
              "3   ham  U dun say so early hor... U c already then say...\n",
              "4   ham  Nah I don't think he goes to usf, he lives aro..."
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4d023fd4-6ee5-4541-95de-6d3afcb61812\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4d023fd4-6ee5-4541-95de-6d3afcb61812')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-4d023fd4-6ee5-4541-95de-6d3afcb61812 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-4d023fd4-6ee5-4541-95de-6d3afcb61812');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ],
      "source": [
        "df.rename(columns={\"v1\":\"label\", \"v2\":\"text\"}, inplace=True)\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "Kt39lBWsbTzn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b72f28cd-ef6d-4a4f-f642-02b7fc951999"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "171"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ],
      "source": [
        "# Finding maximum length of text message\n",
        "\n",
        "np.max(df['text'].apply(lambda x: len(x.split())).values)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ygp6ge--bTzo"
      },
      "source": [
        "<h2>Exploratory data analysis</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "1SHEj4pBbTzo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "241f3554-9645-4386-d5b9-a52942cb138c"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([4825,  747])"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ],
      "source": [
        "# Checking balance of dataset\n",
        "grouped_df = df.groupby('label').count().values.flatten()\n",
        "grouped_df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gXy9BBC0UrZo",
        "outputId": "f7c1520d-0ed5-4553-cc09-7278db129f7f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 5572 entries, 0 to 5571\n",
            "Data columns (total 2 columns):\n",
            " #   Column  Non-Null Count  Dtype \n",
            "---  ------  --------------  ----- \n",
            " 0   label   5572 non-null   object\n",
            " 1   text    5572 non-null   object\n",
            "dtypes: object(2)\n",
            "memory usage: 87.2+ KB\n"
          ]
        }
      ],
      "source": [
        "df.info()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "9fBVFh54UrZo",
        "outputId": "5e3bf908-6185-4e1d-ade8-a94f91727a2f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  label                                               text\n",
              "0   ham  Go until jurong point, crazy.. Available only ...\n",
              "1   ham                      Ok lar... Joking wif u oni...\n",
              "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
              "3   ham  U dun say so early hor... U c already then say...\n",
              "4   ham  Nah I don't think he goes to usf, he lives aro..."
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-84054bbd-9391-402b-aac9-b2bb453e49d6\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-84054bbd-9391-402b-aac9-b2bb453e49d6')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-84054bbd-9391-402b-aac9-b2bb453e49d6 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-84054bbd-9391-402b-aac9-b2bb453e49d6');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ],
      "source": [
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d2LCAR8NbTzo"
      },
      "source": [
        "The 'text' property is a string and must be specified as:\n",
        "\n",
        "      - A string\n",
        "      - A number that will be converted to a string\n",
        "      - A tuple, list, or one-dimensional numpy array \n",
        "      \n",
        "The 'x' and 'y' property is an array that may be specified as a tuple,\n",
        "    list, numpy array, or pandas Series"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "Rsk92WdZbTzo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "outputId": "787973ee-f625-4e3f-f4f7-ba5fa8172662"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.8.3.min.js\"></script>                <div id=\"19e61110-27e1-4538-84f3-dee102bbb53c\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"19e61110-27e1-4538-84f3-dee102bbb53c\")) {                    Plotly.newPlot(                        \"19e61110-27e1-4538-84f3-dee102bbb53c\",                        [{\"marker\":{\"color\":\"#496595\"},\"name\":\"Safe\",\"text\":[\"4825\"],\"textposition\":\"auto\",\"x\":[\"ham\"],\"y\":[4825],\"type\":\"bar\"},{\"marker\":{\"color\":\"#c6ccd8\"},\"name\":\"Spam\",\"text\":[\"747\"],\"textposition\":\"auto\",\"x\":[\"spam\"],\"y\":[747],\"type\":\"bar\"}],                        {\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]],\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]},\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"ternary\":{\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2}}},\"title\":{\"text\":\"Class distribution in the dataset\"}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('19e61110-27e1-4538-84f3-dee102bbb53c');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "fig = go.Figure()\n",
        "\n",
        "fig.add_trace(go.Bar(\n",
        "        x=['ham'],\n",
        "        y=[grouped_df[0]],\n",
        "        name='Safe',\n",
        "        text=[grouped_df[0]],\n",
        "        textposition='auto',\n",
        "        marker_color=pblue\n",
        ")\n",
        "             )\n",
        "fig.add_trace(go.Bar(\n",
        "        x=['spam'],\n",
        "        y=[grouped_df[1]],\n",
        "        name='Spam',\n",
        "        text=[grouped_df[1]],\n",
        "        textposition='auto',\n",
        "        marker_color=pg\n",
        "))\n",
        "\n",
        "fig.update_layout(\n",
        "    title='Class distribution in the dataset')\n",
        "\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "s_u1HbpNbTzo"
      },
      "outputs": [],
      "source": [
        "# Creating series with length as index\n",
        "# Sorting the series by index i.e. length\n",
        "len_df_ham = df[df['label']=='ham'].text.apply(lambda x: len(x.split())).value_counts().sort_index()\n",
        "len_df_spam = df[df['label']=='spam'].text.apply(lambda x: len(x.split())).value_counts().sort_index()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "c_-oTY0hbTzo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "74dafffa-8db3-45be-bbc9-d233042fd59d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1       41\n",
              "2       39\n",
              "3       78\n",
              "4      197\n",
              "5      353\n",
              "      ... \n",
              "103      1\n",
              "109      1\n",
              "125      2\n",
              "162      1\n",
              "171      1\n",
              "Name: text, Length: 82, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ],
      "source": [
        "len_df_ham"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "4LClrcrfbTzp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7e0c6e24-f529-4f7f-c9a7-4280aa345bb1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2      2\n",
              "3      1\n",
              "4      1\n",
              "5      3\n",
              "6      4\n",
              "7      4\n",
              "8      8\n",
              "9      2\n",
              "10     5\n",
              "11     4\n",
              "12    12\n",
              "13    14\n",
              "14     6\n",
              "15    12\n",
              "16     8\n",
              "17    12\n",
              "18    15\n",
              "19    21\n",
              "20    25\n",
              "21    16\n",
              "22    49\n",
              "23    47\n",
              "24    62\n",
              "25    67\n",
              "26    65\n",
              "27    79\n",
              "28    78\n",
              "29    42\n",
              "30    36\n",
              "31    14\n",
              "32    21\n",
              "33     7\n",
              "34     3\n",
              "35     2\n",
              "Name: text, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ],
      "source": [
        "len_df_spam"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "fT6p-eCnbTzp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "outputId": "1ac01d79-0b48-49ac-8528-5fcc56b20f39"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.8.3.min.js\"></script>                <div id=\"6bdea6ae-100d-4fe5-b262-45e9577bbc0e\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"6bdea6ae-100d-4fe5-b262-45e9577bbc0e\")) {                    Plotly.newPlot(                        \"6bdea6ae-100d-4fe5-b262-45e9577bbc0e\",                        [{\"fill\":\"tozeroy\",\"marker\":{\"color\":\"#496595\"},\"name\":\"Safe\",\"x\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,52,53,54,55,56,57,58,59,60,61,62,63,64,66,68,69,70,71,72,76,77,80,88,89,95,96,98,103,109,125,162,171],\"y\":[41,39,78,197,353,417,383,341,295,247,233,184,166,150,149,122,106,112,89,88,73,74,65,72,67,80,81,48,56,59,38,51,35,37,32,19,16,11,11,9,7,5,8,2,3,3,2,3,4,2,4,1,3,1,4,2,3,1,3,4,4,4,2,1,1,1,3,3,1,2,1,1,1,1,1,2,1,1,1,2,1,1],\"type\":\"scatter\"},{\"fill\":\"tozeroy\",\"marker\":{\"color\":\"#c6ccd8\"},\"name\":\"Spam\",\"x\":[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35],\"y\":[2,1,1,3,4,4,8,2,5,4,12,14,6,12,8,12,15,21,25,16,49,47,62,67,65,79,78,42,36,14,21,7,3,2],\"type\":\"scatter\"}],                        {\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]],\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]},\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"ternary\":{\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2}}},\"title\":{\"text\":\"Frequency of SMS lengths\"},\"xaxis\":{\"range\":[0,80]}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('6bdea6ae-100d-4fe5-b262-45e9577bbc0e');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "# X-axis consists of the length of the msgs\n",
        "# Y-axis consists of the frequency of those lengths\n",
        "\n",
        "fig = go.Figure()\n",
        "fig.add_trace(go.Scatter(\n",
        "x=len_df_ham.index,\n",
        "y=len_df_ham.values,\n",
        "name='Safe',\n",
        "fill='tozeroy',\n",
        "marker_color=pblue))\n",
        "\n",
        "fig.add_trace(go.Scatter(\n",
        "x=len_df_spam.index,\n",
        "y=len_df_spam.values,\n",
        "name='Spam',\n",
        "fill='tozeroy',\n",
        "marker_color=pg\n",
        "))\n",
        "\n",
        "fig.update_layout(\n",
        "    title='Frequency of SMS lengths')\n",
        "fig.update_xaxes(range=[0, 80])\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9LwluEfUbTzp"
      },
      "source": [
        "We can see that the safe SMS messages are much shorter than the spam messages."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FH6lVarIbTzp"
      },
      "source": [
        "<h2>Data preprocessing</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "nIEGDD3abTzp"
      },
      "outputs": [],
      "source": [
        "def cleaning(text):\n",
        "    text = str(text).lower()\n",
        "    text = re.sub('\\[.*?\\]', '', text)\n",
        "    text = re.sub('https?://\\S+|www\\.\\S+', '', text)\n",
        "    text = re.sub('<.*?>+', '', text)\n",
        "    text = re.sub('[%s]' % re.escape(string.punctuation), '', text)\n",
        "    text = re.sub('\\n', '', text)\n",
        "    text = re.sub('\\w*\\d\\w*', '', text)\n",
        "    return text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "id": "uMrZll7mbTzp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aac0dda5-6b4e-4648-a5bc-a9960d29b3a4"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0       Go until jurong point, crazy.. Available only ...\n",
              "1                           Ok lar... Joking wif u oni...\n",
              "2       Free entry in 2 a wkly comp to win FA Cup fina...\n",
              "3       U dun say so early hor... U c already then say...\n",
              "4       Nah I don't think he goes to usf, he lives aro...\n",
              "                              ...                        \n",
              "5567    This is the 2nd time we have tried 2 contact u...\n",
              "5568                Will Ì_ b going to esplanade fr home?\n",
              "5569    Pity, * was in mood for that. So...any other s...\n",
              "5570    The guy did some bitching but I acted like i'd...\n",
              "5571                           Rofl. Its true to its name\n",
              "Name: text, Length: 5572, dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ],
      "source": [
        "df['text']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "pjA1Mca9bTzp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "38e50faf-089f-4bae-fc7a-d0ed78a3152c"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0       go until jurong point crazy available only in ...\n",
              "1                                 ok lar joking wif u oni\n",
              "2       free entry in  a wkly comp to win fa cup final...\n",
              "3             u dun say so early hor u c already then say\n",
              "4       nah i dont think he goes to usf he lives aroun...\n",
              "                              ...                        \n",
              "5567    this is the  time we have tried  contact u u h...\n",
              "5568                  will ì b going to esplanade fr home\n",
              "5569    pity  was in mood for that soany other suggest...\n",
              "5570    the guy did some bitching but i acted like id ...\n",
              "5571                            rofl its true to its name\n",
              "Name: text, Length: 5572, dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ],
      "source": [
        "df['text'] = df['text'].apply(cleaning)\n",
        "df['text']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CIReUaWfUrZp",
        "outputId": "9ac7a67f-be80-4ab2-c777-4134c1e62f72"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ],
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "id": "9Q4qZD12bTzp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "925b3abd-de5a-483e-e94e-6fa0caab5c81"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0       go jurong point crazy available bugis n great ...\n",
              "1                                   ok lar joking wif oni\n",
              "2       free entry  wkly comp win fa cup final tkts  m...\n",
              "3                           dun say early hor already say\n",
              "4             nah dont think goes usf lives around though\n",
              "                              ...                        \n",
              "5567     time tried  contact å£ pound prize  claim eas...\n",
              "5568                          ì b going esplanade fr home\n",
              "5569                         pity  mood soany suggestions\n",
              "5570    guy bitching acted like id interested buying s...\n",
              "5571                                       rofl true name\n",
              "Name: text, Length: 5572, dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ],
      "source": [
        "# Removing stop words\n",
        "stop_words = stopwords.words('english')\n",
        "more = ['u', 'im', 'c']\n",
        "stop_words = stop_words + more\n",
        "\n",
        "\n",
        "def sw_rem(text):\n",
        "    text = ' '.join(word for word in text.split(' ') if word not in stop_words)\n",
        "    return text\n",
        "\n",
        "df['text'] = df['text'].apply(sw_rem)\n",
        "df['text']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7mtqTbkZbTzp"
      },
      "source": [
        "<h2>Stemming and lemmatization</h2>\n",
        "\n",
        "Documents and other forms of text use different forms of the same words, such as play, playing, played. There are families of derivationally related words that have similar meanings. Our main task with stemming and lemmatization is to reduce all these derived words into the parent/family word, therefore reducing the total vocabulary while retaining information.\n",
        "\n",
        "* **Stemming** - Omits the ends of words to achieve the goal correctly, this works **most of the times** and can also remove the derivational suffix\n",
        "\n",
        "* **Lemmatization** - Working with a vocabulary and morphological analysis of wrods, removing inflectional endings only and returning the base and dictionary form of a word.\n",
        "\n",
        "As we do not require much emphasis on words, we will focus more on stemming than lemmatization,.\n",
        "\n",
        "<h3>Stemming algorithms</h3>\n",
        "\n",
        "We have multiple algorithms to achieve our stemming goals, some of them are as follows:\n",
        "\n",
        "* PorterStemmer - Fast and efficient. Strips off the end (suffix) to produce the stems. It does not follow linguistics but rather a set of 05 rules for diferent cases. \n",
        "\n",
        "* SnowballStemmer - Generate a set of rules for any language. These are useful for non-english stemming tasks.\n",
        "\n",
        "* LancasterStemmer - Iterative algorithm, uses about 120 rules, it tries to find an applicable rule by the last character of each word. The last character may be omitted or replaced."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "id": "sIVQJkvabTzq"
      },
      "outputs": [],
      "source": [
        "stems = nltk.SnowballStemmer('english')\n",
        "\n",
        "def stemming(text):\n",
        "    text = ' '.join(stems.stem(word) for word in text.split())\n",
        "    return text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "id": "FMajbaLCbTzq",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "027b8ba3-1ce4-4a8d-b459-bc113250ed42"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  label                                               text\n",
              "0   ham  go jurong point crazi avail bugi n great world...\n",
              "1   ham                                ok lar joke wif oni\n",
              "2  spam  free entri wkli comp win fa cup final tkts may...\n",
              "3   ham                      dun say earli hor alreadi say\n",
              "4   ham          nah dont think goe usf live around though"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8144f23b-a1d1-422a-90c2-912587ebb8dd\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>go jurong point crazi avail bugi n great world...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>ok lar joke wif oni</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>free entri wkli comp win fa cup final tkts may...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>dun say earli hor alreadi say</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>nah dont think goe usf live around though</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8144f23b-a1d1-422a-90c2-912587ebb8dd')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-8144f23b-a1d1-422a-90c2-912587ebb8dd button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-8144f23b-a1d1-422a-90c2-912587ebb8dd');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ],
      "source": [
        "df['text'] = df['text'].apply(stemming)\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "id": "ZmOR_yEabTzq"
      },
      "outputs": [],
      "source": [
        "# Creating a pipeline\n",
        "\n",
        "def pipeline(text):\n",
        "    text = cleaning(text)\n",
        "    text = ' ' .join(word for word in text.split(' ') if word not in stop_words)\n",
        "    text = ' '.join(stems.stem(word) for word in text.split(' '))\n",
        "    return text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "id": "lTq5zuBQbTzq",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "acea6cab-4e54-4b2e-e87a-0bb62e7e99cd"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  label                                               text\n",
              "0   ham  go jurong point crazi avail bugi n great world...\n",
              "1   ham                                ok lar joke wif oni\n",
              "2  spam  free entri wkli comp win fa cup final tkts may...\n",
              "3   ham                        dun say ear hor alreadi say\n",
              "4   ham          nah dont think goe usf live around though"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3cc8d505-e602-4043-a2b7-49e66c2078c1\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>go jurong point crazi avail bugi n great world...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>ok lar joke wif oni</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>free entri wkli comp win fa cup final tkts may...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>dun say ear hor alreadi say</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>nah dont think goe usf live around though</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3cc8d505-e602-4043-a2b7-49e66c2078c1')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-3cc8d505-e602-4043-a2b7-49e66c2078c1 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-3cc8d505-e602-4043-a2b7-49e66c2078c1');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ],
      "source": [
        "df['text'] = df['text'].apply(pipeline)\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "id": "PLYJUYy7bTzq",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "e74c8d64-35e6-44ef-9b4e-fa6537a7a8bb"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  label                                               text  label_num\n",
              "0   ham  go jurong point crazi avail bugi n great world...          0\n",
              "1   ham                                ok lar joke wif oni          0\n",
              "2  spam  free entri wkli comp win fa cup final tkts may...          1\n",
              "3   ham                        dun say ear hor alreadi say          0\n",
              "4   ham          nah dont think goe usf live around though          0"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d8e255df-9b07-4013-9b03-e84bab6ff6fb\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "      <th>label_num</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>go jurong point crazi avail bugi n great world...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>ok lar joke wif oni</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>free entri wkli comp win fa cup final tkts may...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>dun say ear hor alreadi say</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>nah dont think goe usf live around though</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d8e255df-9b07-4013-9b03-e84bab6ff6fb')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d8e255df-9b07-4013-9b03-e84bab6ff6fb button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d8e255df-9b07-4013-9b03-e84bab6ff6fb');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ],
      "source": [
        "# Encoding the categorical target variable\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "\n",
        "le = LabelEncoder()\n",
        "le.fit(df['label'])\n",
        "\n",
        "df['label_num'] = le.transform(df['label'])\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Me7mobO4bTzq"
      },
      "source": [
        "<h2>Visualizing tokens</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {
        "id": "U0dml2-qbTzq"
      },
      "outputs": [],
      "source": [
        "# This will combine all the text values for safe sms\n",
        "#' '.join(text for text in df[df['label']=='ham'].text)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from PIL import Image\n",
        "import requests\n",
        "from io import BytesIO\n",
        "\n",
        "url1 = 'https://raw.githubusercontent.com/Sahel-Eskandar/DataScience-Projects/main/Data/upvote.png'\n",
        "\n",
        "response1 = requests.get(url1)\n",
        "img = Image.open(BytesIO(response1.content))"
      ],
      "metadata": {
        "id": "M1W4w8YJenPc"
      },
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {
        "id": "GEaBZ0Z0bTzq",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 597
        },
        "outputId": "54974ec4-2a5c-4a6a-96be-e746f1e015c3"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1080x720 with 1 Axes>"
            ],
            "image/png": "[encoded data, 276140 characters]\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# Extracting the twitter word cloud mask\n",
        "upvote_mask = np.array(img)\n",
        "wc = WordCloud(background_color='white', max_words=200, mask=upvote_mask)\n",
        "\n",
        "wc.generate(' '.join(text for text in df[df['label']=='ham'].text))\n",
        "plt.figure(figsize=(15, 10))\n",
        "plt.title('Top words for safe messages', fontdict={'size':22})\n",
        "plt.imshow(wc)\n",
        "plt.axis('off')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "url2 = 'https://raw.githubusercontent.com/Sahel-Eskandar/DataScience-Projects/main/Data/dislike.png'\n",
        "\n",
        "response2 = requests.get(url2)\n",
        "img2 = Image.open(BytesIO(response2.content))"
      ],
      "metadata": {
        "id": "Yzn2WiXnfIwg"
      },
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {
        "id": "efHuklgJbTzq",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 597
        },
        "outputId": "60104f5e-f1dd-4d98-ed0f-fa1a5fdd6bbe"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1080x720 with 1 Axes>"
            ],
            "image/png": "[encoded data, 276740 characters]\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# Extracting the twitter word cloud mask\n",
        "dislike_mask = np.array(img2)\n",
        "wc = WordCloud(background_color='white', max_words=200, mask=dislike_mask)\n",
        "\n",
        "wc.generate(' '.join(text for text in df[df['label']=='spam'].text))\n",
        "plt.figure(figsize=(15, 10))\n",
        "plt.title('Top words for Spam messages', fontdict={'size':22})\n",
        "plt.imshow(wc)\n",
        "plt.axis('off')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HHQbjdNubTzq"
      },
      "source": [
        "<h2>Vectorization</h2>\n",
        "\n",
        "We currently have each text record in string format. We need to convert each of those records into a vector that our models can work with. We will first do this using the bag-of-words model.\n",
        "\n",
        "We will use two major approaches here\n",
        "\n",
        "* **CountVectorizer** - Working on frequency of each word in the given string.\n",
        "\n",
        "* **Term frequency-inverse document frqeuency TFIDF** - Works on frequency divided by the appearance of the given word in the total documents."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "id": "2H_Q-onxbTzq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "67836da0-c485-46d8-f015-e79e417e092a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(5572, 5572)"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ],
      "source": [
        "x = df['text']\n",
        "y = df['label_num']\n",
        "\n",
        "len(x), len(y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "metadata": {
        "id": "wmhFTemSbTzr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3f334216-a49a-4426-d9de-4af7193e9dfc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "4179 4179\n",
            "1393 1393\n"
          ]
        }
      ],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "x_train, x_test, y_train, y_test = train_test_split(x, y)\n",
        "print(len(x_train), len(y_train))\n",
        "print(len(x_test), len(y_test))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "metadata": {
        "id": "CZeSfKTDbTzr"
      },
      "outputs": [],
      "source": [
        "# First working with count vectorizer\n",
        "\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "# instantiate the vectorizer\n",
        "count = CountVectorizer()\n",
        "count.fit(x)\n",
        "\n",
        "x_train_num = count.transform(x_train)\n",
        "x_test_num = count.transform(x_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GGfRWGYWbTzr"
      },
      "source": [
        "The CountVectorizer model can be tuned in a variety of ways:\n",
        "\n",
        "* Stop words - Extremely common words can be omitted by the model by setting this parameter to the language corresponding to the text.\n",
        "\n",
        "* ngram_range - It pairs up words together as features. If we consider bigrams and we have a sentence \"I am happy\", we will have two features - [\"I am\", \"am happy\"]. We can define a range of ngrams, so if we have the same sentence with a range from 1 to 2, our features will be:  `[\"I\", \"am\", \"happy\", \"I am\", \"am happy\"]`. This increase is features helps to fine tune the model.\n",
        "\n",
        "* min_df, max_df - Minimum and maximum frequencies of words of n-grams that can be used as features. If either of the conditions are not met, the feature will be omitted.\n",
        "\n",
        "* max_features - Choose the most frequent words and drop everything else."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "metadata": {
        "id": "hpaGBv5xbTzr"
      },
      "outputs": [],
      "source": [
        "# Example of a tuned model\n",
        "count_tuned = CountVectorizer(stop_words='english', ngram_range=(1,2), min_df=0.1, max_df=0.7, max_features=100)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "metadata": {
        "id": "y2_h-DwobTzr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bb9b0f96-8148-4449-e994-8fed3e142090"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<4179x6696 sparse matrix of type '<class 'numpy.float64'>'\n",
              "\twith 32374 stored elements in Compressed Sparse Row format>"
            ]
          },
          "metadata": {},
          "execution_count": 56
        }
      ],
      "source": [
        "# Working with TF-IDF now\n",
        "from sklearn.feature_extraction.text import TfidfTransformer\n",
        "# We are using transformer here\n",
        "# If we use vectorizer, we can directly use the text\n",
        "tfidf = TfidfTransformer()\n",
        "\n",
        "tfidf.fit(x_train_num)\n",
        "x_train_tfidf = tfidf.transform(x_train_num)\n",
        "\n",
        "x_train_tfidf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "stMzLo3NbTzr"
      },
      "source": [
        "<h2>Working with Embeddings - GloVe</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "metadata": {
        "id": "cz2aUBrVbTzr"
      },
      "outputs": [],
      "source": [
        "text = df['text']\n",
        "label = df['label_num']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "metadata": {
        "id": "Ke4YuaK3bTzr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2b16031e-b805-46b8-d267-c4f4b0664a19"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "6724"
            ]
          },
          "metadata": {},
          "execution_count": 58
        }
      ],
      "source": [
        "# Calculating the total vocabulary\n",
        "tk = Tokenizer()\n",
        "tk.fit_on_texts(text)\n",
        "\n",
        "vocab = len(tk.word_index)+1\n",
        "vocab"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-7Rd4foTbTzr"
      },
      "source": [
        "Now we will proceed with converting the text to numerical values and also padding the vectors so each of them are of equal length. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {
        "id": "BnSQtL-WbTzr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c26315af-7734-4b32-ea82-295013961b4d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "80"
            ]
          },
          "metadata": {},
          "execution_count": 59
        }
      ],
      "source": [
        "# Maximum length\n",
        "max_len = np.max(df['text'].apply(lambda x: len(x.split())).values)\n",
        "max_len"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "metadata": {
        "id": "gaCkx6MxbTzr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "68ae242e-07fd-4053-9173-28250d325406"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0       go jurong point crazi avail bugi n great world...\n",
              "1                                     ok lar joke wif oni\n",
              "2       free entri wkli comp win fa cup final tkts may...\n",
              "3                             dun say ear hor alreadi say\n",
              "4               nah dont think goe usf live around though\n",
              "                              ...                        \n",
              "5567    time tri contact å£ pound prize claim easi cal...\n",
              "5568                              ì b go esplanad fr home\n",
              "5569                              piti mood soani suggest\n",
              "5570    guy bitch act like id interest buy someth el n...\n",
              "5571                                       rofl true name\n",
              "Name: text, Length: 5572, dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 60
        }
      ],
      "source": [
        "text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "metadata": {
        "id": "5TT47dLwbTzr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6cf9b600-a91c-4ad4-c5e6-af1ac6798e86"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[   2, 3180,  274, ...,    0,    0,    0],\n",
              "       [   8,  236,  527, ...,    0,    0,    0],\n",
              "       [   9,  356,  588, ...,    0,    0,    0],\n",
              "       ...,\n",
              "       [6722, 1002, 6723, ...,    0,    0,    0],\n",
              "       [ 138, 1251, 1603, ...,    0,    0,    0],\n",
              "       [1987,  378,  170, ...,    0,    0,    0]], dtype=int32)"
            ]
          },
          "metadata": {},
          "execution_count": 61
        }
      ],
      "source": [
        "def embedding(text):\n",
        "    return tk.texts_to_sequences(text)\n",
        "\n",
        "train_padded = pad_sequences(embedding(text), 80, padding='post')\n",
        "train_padded"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7rthEL5XbTzs"
      },
      "source": [
        "<h2>GloVe Embeddings</h2>\n",
        "\n",
        "These embeddings are based on the principle that we can derive sematic relationships between words from their co-occurence matrix. This embedding focuses on words co-occurrences over the whole corpus. \n",
        "\n",
        "They are a form of word representation that try to merge human understanding of languages into their structure. They have a learned representation in an n-dimension space, where words with similar meanings have similar embeddings. Two similar words are represented by almost similar vectors that are at a small distance in the vector space.\n",
        "\n",
        "When using a vector space, all the words are represented as vectors in a predefined N-dimension vector space. Each word is mapped to a vector and the vector values are learned in a way that resembles a neural network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DN6dVnDebTzs"
      },
      "outputs": [],
      "source": [
        "# Using our helper functions for GloVe\n",
        "\n",
        "embedding_dict = dict()\n",
        "embedding_dim = 100\n",
        "\n",
        "# Each word is represented in one line in the text file\n",
        "# Format - Word val1 val2 val3......val-n for n-dimension vector space\n",
        "\n",
        "with open('../data/glove.6B.100d.txt') as fp:\n",
        "    for line in fp.readlines():\n",
        "        records = line.split()\n",
        "        word = records[0]\n",
        "        vector = np.asarray(records[1:], dtype='float32')\n",
        "        embedding_dict[word] = vector"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lCbG_BxLbTzs"
      },
      "outputs": [],
      "source": [
        "# Creating a matrix for each word as index (word numerical value extracted from tokenizer\n",
        "# with N-features (corresponding to GloVe)\n",
        "# We will replace the matrix elements by the words and their embeddings\n",
        "\n",
        "# Our embeddings will also consist embeddings for padding\n",
        "embedding_matrix = np.zeros((vocab, embedding_dim))\n",
        "\n",
        "for word, index in tk.word_index.items():\n",
        "    embed_vector = embedding_dict.get(word)\n",
        "    if embed_vector is not None:\n",
        "        embedding_matrix[index] = embed_vector\n",
        "        \n",
        "embedding_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jzkSV1dxbTzs"
      },
      "outputs": [],
      "source": [
        "# We will be creating seaborn and plotly confusion matrices\n",
        "import plotly.figure_factory as ff\n",
        "x_axes = ['Safe','Spam']\n",
        "y_axes = ['Spam', 'Safe']\n",
        "\n",
        "def conf_matrix(z, x=x_axes, y=y_axes):\n",
        "    z = np.flip(z, 0)\n",
        "    # Change each element of z to string \n",
        "    # This allows them to be used as annotations\n",
        "    z_str = [[str(y) for y in x] for x in z]\n",
        "    fig = ff.create_annotated_heatmap(z, x=x, y=y, annotation_text=z_str)\n",
        "    \n",
        "    fig.update_layout(title_text='Confusion matrix', xaxis=dict(title='Predicted Value'),\n",
        "                     yaxis=dict(title='Real value'))\n",
        "    \n",
        "    fig['data'][0]['showscale'] = True\n",
        "    return fig"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Vrx_tFXIbTzs"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import confusion_matrix\n",
        "categories=['Safe', 'Spam']\n",
        "def seaborn_conf(y, ypred):\n",
        "    y_true = [\"Safe\", \"Spam\"]\n",
        "    y_pred = [\"Safe\", \"Spam\"]\n",
        "    cf = confusion_matrix(y, ypred)\n",
        "    df_cm = pd.DataFrame(cf, columns=np.unique(y_true), index = np.unique(y_true))\n",
        "    plt.figure(figsize=(8,6))\n",
        "    sns.heatmap(df_cm, annot=True, fmt='g')\n",
        "    plt.title('Confusion matrix')\n",
        "    plt.xlabel('Predicted value')\n",
        "    plt.ylabel('Real value')\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jCJ0ZULybTzs"
      },
      "source": [
        "<h2>Model creation and prediction</h2>\n",
        "\n",
        "We will first start with the **naive bayes classifier** which comes from a family of simple \"probabilistic classifiers\" based on application of Bayes theroem with strong independent assumptions between features.\n",
        "\n",
        "The model is highly scalable, with number of parameters being linear with number of variables. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SzBNsBkFbTzs"
      },
      "outputs": [],
      "source": [
        "from sklearn.naive_bayes import MultinomialNB\n",
        "nb = MultinomialNB()\n",
        "\n",
        "# Train the model - CountVectorizer model\n",
        "nb.fit(x_train_num, y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FGhn_HVSbTzs"
      },
      "outputs": [],
      "source": [
        "# Class and probability predictions\n",
        "yp_class = nb.predict(x_test_num)\n",
        "yp_prob = nb.predict_proba(x_test_num)[:, 1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ClNtr8a4bTzs"
      },
      "outputs": [],
      "source": [
        "from sklearn import metrics\n",
        "print(metrics.accuracy_score(y_test, yp_class))\n",
        "seaborn_conf(y_test, yp_class)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ikZX1rnObTzs"
      },
      "outputs": [],
      "source": [
        "metrics.roc_auc_score(y_test, yp_prob)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bl6Fl4dnbTzs"
      },
      "source": [
        "<h2>Working with Naive Bayes + TF-IDF</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RvYYbgVcbTzs"
      },
      "outputs": [],
      "source": [
        "from sklearn.feature_extraction.text import TfidfTransformer\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "pipe = Pipeline([('bow', CountVectorizer()), \n",
        "                 ('tfid', TfidfTransformer()),  \n",
        "                 ('model', MultinomialNB())])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8qADl2TTbTzs"
      },
      "outputs": [],
      "source": [
        "pipe.fit(x_train, y_train)\n",
        "yp_class = pipe.predict(x_test)\n",
        "print(metrics.accuracy_score(y_test, yp_class))\n",
        "seaborn_conf(y_test, yp_class)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4XI7iV7bbTzt"
      },
      "source": [
        "<h2>XGBoost</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [],
        "id": "AMpF3sBFUrZs"
      },
      "outputs": [],
      "source": [
        "!pip install xgboost"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uPISmiZsbTzt"
      },
      "outputs": [],
      "source": [
        "import xgboost as xgb\n",
        "pipe = Pipeline([\n",
        "    ('bow', CountVectorizer()), \n",
        "    ('tfid', TfidfTransformer()),  \n",
        "    ('model', xgb.XGBClassifier(\n",
        "        learning_rate=0.1,\n",
        "        max_depth=6,\n",
        "        n_estimators=90,\n",
        "        use_label_encoder=False,\n",
        "        eval_metric='auc'\n",
        "    ))\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UDCumywBbTzt"
      },
      "outputs": [],
      "source": [
        "pipe.fit(x_train, y_train)\n",
        "yp_class_test = pipe.predict(x_test)\n",
        "yp_class_train = pipe.predict(x_train)\n",
        "\n",
        "print('Training accuracy score: {}'.format(metrics.accuracy_score(y_train, yp_class_train)))\n",
        "print('Testing accuracy score: {}'.format(metrics.accuracy_score(y_test, yp_class_test)))\n",
        "\n",
        "seaborn_conf(y_test, yp_class_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nLVWMr-bbTzt"
      },
      "source": [
        "<h2>LSTMs and GloVE embeddings</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mMroy8EwbTzt"
      },
      "outputs": [],
      "source": [
        "train_padded.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eg7rW6vQbTzt"
      },
      "outputs": [],
      "source": [
        "x_train, x_test, y_train, y_test = train_test_split(train_padded, label, test_size=0.2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_zSu_ACsbTzt"
      },
      "outputs": [],
      "source": [
        "model = Sequential()\n",
        "model.add(Embedding(input_dim=embedding_matrix.shape[0], \n",
        "                   output_dim=embedding_matrix.shape[1],\n",
        "                   weights=[embedding_matrix],\n",
        "                   input_length=max_len\n",
        "                   )\n",
        "         )\n",
        "model.add(Bidirectional(LSTM(max_len, return_sequences=True, recurrent_dropout=0.15)))\n",
        "model.add(GlobalMaxPool1D())\n",
        "model.add(BatchNormalization())\n",
        "model.add(Dropout(0.3))\n",
        "model.add(Dense(max_len, activation='relu'))\n",
        "model.add(Dropout(0.3))\n",
        "model.add(Dense(max_len, activation='relu'))\n",
        "model.add(Dropout(0.3))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "\n",
        "\n",
        "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2WgO2yjxbTzt"
      },
      "outputs": [],
      "source": [
        "# Defining Callbacks\n",
        "# Checkpoints in case our model stops training due to some circumstance - saving progress\n",
        "checkpoints = ModelCheckpoint('ck_model.h5', monitor='val_loss', verbose=1, save_best_only=True)\n",
        "# Reducing the learning rate if no improvement in validation loss over 5 epochs\n",
        "# This is to train the model better\n",
        "reduce_lr = ReduceLROnPlateau(monitor='val_loss', vactor=0.1, verbose=1, patience=5, min_lr=0.0001)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2yPzfX0ubTzt"
      },
      "outputs": [],
      "source": [
        "history = model.fit(x_train, y_train, epochs=10, batch_size=32, validation_data=(x_test, y_test), verbose=1, callbacks=[reduce_lr, checkpoints])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KanoH59zbTzt"
      },
      "outputs": [],
      "source": [
        "# Plotting the results\n",
        "def learning_curve(history, arr):\n",
        "    fig, ax=plt.subplots(1, 2, figsize=(20, 5))\n",
        "    for idx in range(2):\n",
        "        ax[idx].plot(history.history[arr[idx][0]])\n",
        "        ax[idx].plot(history.history[arr[idx][1]])\n",
        "        ax[idx].legend([arr[idx][0], arr[idx][1]])\n",
        "        ax[idx].set_xlabel('Epochs')\n",
        "        ax[idx].set_ylabel('Value')\n",
        "        ax[idx].set_title(arr[idx][0]+' X '+ arr[idx][1])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OCM5efEGbTzt"
      },
      "outputs": [],
      "source": [
        "learning_curve(history, [['loss', 'val_loss'], ['accuracy', 'val_accuracy']])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D1URS30kbTzt"
      },
      "outputs": [],
      "source": [
        "yp = model.predict(x_test)\n",
        "yp"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jBdGjLRHbTzt"
      },
      "outputs": [],
      "source": [
        "yp = (model.predict(x_test)>0.5).astype('int32')\n",
        "yp"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r4FwSEr4bTzu"
      },
      "outputs": [],
      "source": [
        "seaborn_conf(y_test, yp)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8rIqqPvcbTzu"
      },
      "source": [
        "<h2>Transformers - BERT</h2>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aLT-LVM2bTzu"
      },
      "source": [
        "BERT has revolutionized the world of NLP by providing state-of-the-art results on many NLP tasks. BERT stands for Bidirectional Encoder Representation from Transformer. It is the state-of-the-art embedding model published by Google. It has created a major breakthrough in the field of NLP by providing greater results in many NLP tasks, such as question answering, text generation, sentence classification, and many more besides. One of the major reasons for the success of BERT is that it is a context-based embedding model, unlike other popular embedding models, such as word2vec, which are context-free."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Uf4vgWgjbTzu"
      },
      "outputs": [],
      "source": [
        "!pip install transformers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "znGH9meZbTzu"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers import Dense, Input\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "from tensorflow.keras.models import Model\n",
        "from tensorflow.keras.callbacks import ModelCheckpoint\n",
        "\n",
        "import transformers\n",
        "from tqdm.notebook import tqdm\n",
        "from tokenizers import BertWordPieceTokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aGokzdJ9bTzu"
      },
      "outputs": [],
      "source": [
        "from transformers import BertTokenizer\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')\n",
        "\n",
        "def bert_encode(data, maximum_length):\n",
        "    input_ids=[]\n",
        "    attention_masks=[]\n",
        "    for text in data:\n",
        "        encoded = tokenizer.encode_plus(text, add_special_tokens=True, max_length = maximum_length, pad_to_max_length=True, return_attention_mask=True)\n",
        "        input_ids.append(encoded['input_ids'])\n",
        "        attention_masks.append(encoded['attention_mask'])\n",
        "    return np.array(input_ids), np.array(attention_masks)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EU2JCaKIbTzu"
      },
      "outputs": [],
      "source": [
        "np.max(df['text'].apply(lambda x: len(x.split())).values)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5gVUFY2TbTzu"
      },
      "outputs": [],
      "source": [
        "bt_text = df['text']\n",
        "bt_label = df['label_num']\n",
        "\n",
        "bt_ids, bt_masks = bert_encode(bt_text, 80)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nv9Vcr15bTzu"
      },
      "outputs": [],
      "source": [
        "from transformers import TFBertModel\n",
        "def create_model(bert_model):\n",
        "    \n",
        "    input_ids = tf.keras.Input(shape=(80,),dtype='int32')\n",
        "    attention_masks = tf.keras.Input(shape=(80,),dtype='int32')\n",
        "\n",
        "    output = bert_model([input_ids,attention_masks])\n",
        "    output = output[1]\n",
        "    output = tf.keras.layers.Dense(32,activation='relu')(output)\n",
        "    output = tf.keras.layers.Dropout(0.2)(output)\n",
        "    output = tf.keras.layers.Dense(1,activation='sigmoid')(output)\n",
        "    \n",
        "    model = tf.keras.models.Model(inputs = [input_ids,attention_masks],outputs = output)\n",
        "    model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=['accuracy'])\n",
        "    return model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lFBNx-OnbTzu"
      },
      "outputs": [],
      "source": [
        "bert_model = TFBertModel.from_pretrained('bert-base-uncased')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oUQ0B9JDbTzu"
      },
      "outputs": [],
      "source": [
        "model = create_model(bert_model)\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2hr4E8LrbTzu"
      },
      "outputs": [],
      "source": [
        "history = model.fit([bt_ids, bt_masks], bt_label, validation_split=0.25, epochs=3, batch_size=10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jwi_PISCbTzu"
      },
      "outputs": [],
      "source": [
        "learning_curve(history, [['loss', 'val_loss'],['accuracy', 'val_accuracy']])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uNyb4y6sbTzv"
      },
      "source": [
        "<h2>Working with disaster tweets - Dataset</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v3uw3v5bbTzv"
      },
      "outputs": [],
      "source": [
        "df = pd.read_csv('../input/nlp-getting-started/train.csv')\n",
        "df_test = pd.read_csv('../input/nlp-getting-started/test.csv')\n",
        "\n",
        "df_train = df_train.dropna(axis=1)\n",
        "df_train.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JJcC1IWgbTzv"
      },
      "outputs": [],
      "source": [
        "df.groupby('target').count()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uc1p_iP7bTzv"
      },
      "outputs": [],
      "source": [
        "# We can use .agg('count').values also\n",
        "class_counts = df.groupby('target').id.count().values\n",
        "class_counts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SBrm21-6bTzv"
      },
      "outputs": [],
      "source": [
        "fig = go.Figure()\n",
        "\n",
        "fig.add_trace(go.Bar(\n",
        "        x=['Fake disaster'],\n",
        "        y=[class_counts[0]],\n",
        "        name='Fake',\n",
        "        text=[class_counts[0]],\n",
        "        textposition='auto',\n",
        "        marker_color=pblue\n",
        ")\n",
        "             )\n",
        "fig.add_trace(go.Bar(\n",
        "        x=['Real disaster'],\n",
        "        y=[class_counts[1]],\n",
        "        name='Real',\n",
        "        text=[class_counts[1]],\n",
        "        textposition='auto',\n",
        "        marker_color=pg\n",
        "))\n",
        "\n",
        "fig.update_layout(\n",
        "    title='Class distribution in the dataset')\n",
        "\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r4e5184YbTzv"
      },
      "outputs": [],
      "source": [
        "len_real = df[df['target']==1].text.apply(lambda x: len(x.split())).value_counts().sort_index()\n",
        "len_fake = df[df['target']==0].text.apply(lambda x: len(x.split())).value_counts().sort_index()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4Kj8G-pvbTzv"
      },
      "outputs": [],
      "source": [
        "fig = go.Figure()\n",
        "fig.add_trace(go.Scatter(\n",
        "    x=len_real.index,\n",
        "    y=len_real.values,\n",
        "    name='Real disaster',\n",
        "    fill='tozeroy',\n",
        "    marker_color=pblue,\n",
        "))\n",
        "fig.add_trace(go.Scatter(\n",
        "    x=len_fake.index,\n",
        "    y=len_fake.values,\n",
        "    name='Fake disaster',\n",
        "    fill='tozeroy',\n",
        "    marker_color=pg,\n",
        "))\n",
        "fig.update_layout(\n",
        "    title='<span style=\"font-size:32px; font-family:Times New Roman\">Data Roles in Different Fields</span>'\n",
        ")\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UasL5kOibTzv"
      },
      "source": [
        "<h2>Data pre-processing and cleaning</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DIzgUwnebTzv"
      },
      "outputs": [],
      "source": [
        "def remove_url(text):\n",
        "    url = re.compile(r'https?://\\S+|www\\.\\S+')\n",
        "    return url.sub(r'', text)\n",
        "\n",
        "\n",
        "def remove_emoji(text):\n",
        "    emoji_pattern = re.compile(\n",
        "        '['\n",
        "        u'\\U0001F600-\\U0001F64F'  # emoticons\n",
        "        u'\\U0001F300-\\U0001F5FF'  # symbols & pictographs\n",
        "        u'\\U0001F680-\\U0001F6FF'  # transport & map symbols\n",
        "        u'\\U0001F1E0-\\U0001F1FF'  # flags (iOS)\n",
        "        u'\\U00002702-\\U000027B0'\n",
        "        u'\\U000024C2-\\U0001F251'\n",
        "        ']+',\n",
        "        flags=re.UNICODE)\n",
        "    return emoji_pattern.sub(r'', text)\n",
        "\n",
        "\n",
        "def remove_html(text):\n",
        "    html = re.compile(r'<.*?>|&([a-z0-9]+|#[0-9]{1,6}|#x[0-9a-f]{1,6});')\n",
        "    return re.sub(html, '', text)\n",
        "\n",
        "def clean_text(text):\n",
        "    '''Make text lowercase, remove text in square brackets,remove links,remove punctuation\n",
        "    and remove words containing numbers.'''\n",
        "    text = str(text).lower()\n",
        "    text = re.sub('\\[.*?\\]', '', text)\n",
        "    text = re.sub(\n",
        "        'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', \n",
        "        '', \n",
        "        text\n",
        "    )\n",
        "    text = re.sub('https?://\\S+|www\\.\\S+', '', text)\n",
        "    text = re.sub('<.*?>+', '', text)\n",
        "    text = re.sub('[%s]' % re.escape(string.punctuation), '', text)\n",
        "    text = re.sub('\\n', '', text)\n",
        "    text = re.sub('\\w*\\d\\w*', '', text)\n",
        "    \n",
        "    text = remove_url(text)\n",
        "    text = remove_emoji(text)\n",
        "    text = remove_html(text)\n",
        "    \n",
        "    return text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0SCQj4C0bTzv"
      },
      "outputs": [],
      "source": [
        "# Testing the function\n",
        "remove_emoji(\"Omg another Earthquake 😔😔\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J8tD8IjLbTzv"
      },
      "outputs": [],
      "source": [
        "stopw = stopwords.words('english')\n",
        "more = ['u', 'im', 'c']\n",
        "stopw = stopw + more\n",
        "\n",
        "stemmer = nltk.SnowballStemmer('english')\n",
        "\n",
        "def data_cleaning(text):\n",
        "    text = clean_text(text)\n",
        "    text = ' '.join(stemmer.stem(word) for word in text.split(' ') if word not in stopw)\n",
        "    return text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BMvdTaUqbTzw"
      },
      "outputs": [],
      "source": [
        "df['cleaned_text'] = df['text'].apply(data_cleaning)\n",
        "df_test['cleaned_text'] = df_test['text'].apply(data_cleaning)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "evHQRSjubTzw"
      },
      "outputs": [],
      "source": [
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YImcqD1ebTzw"
      },
      "source": [
        "<h2>WordCloud analysis</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UlTNh6cibTzw"
      },
      "outputs": [],
      "source": [
        "def corpus(df, label):\n",
        "    corpus=[]\n",
        "    for x in df[df['target']==label]['cleaned_text'].str.split():\n",
        "        for i in x:\n",
        "            corpus.append(i)\n",
        "    return corpus"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TAvzb2_WbTzw"
      },
      "outputs": [],
      "source": [
        "corpus_reald = corpus(df, 1)\n",
        "dic = defaultdict(int)\n",
        "\n",
        "# Creating a dictionary with frequency of words\n",
        "for word in corpus_reald:\n",
        "    dic[word]+=1\n",
        "    \n",
        "# Sorting words by descending frequency\n",
        "top = sorted(dic.items(), key=lambda x: x[1], reverse=True)[:10]\n",
        "top"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aO1RqmmRbTzw"
      },
      "outputs": [],
      "source": [
        "twitter_mask = np.array(Image.open('/kaggle/input/masksforwordclouds/twitter_mask3.jpg'))\n",
        "\n",
        "wc = WordCloud(\n",
        "    background_color='white', \n",
        "    max_words=200, \n",
        "    mask=twitter_mask,\n",
        ")\n",
        "wc.generate(' '.join(text for text in df.loc[df['target'] == 1, 'cleaned_text']))\n",
        "plt.figure(figsize=(18,10))\n",
        "plt.title('Wordcloud for real disasters', \n",
        "          fontdict={'size': 22,  'verticalalignment': 'bottom'})\n",
        "plt.imshow(wc)\n",
        "plt.axis(\"off\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OM1U6i4ubTzw"
      },
      "outputs": [],
      "source": [
        "corpus_faked = corpus(df, 0)\n",
        "dic = defaultdict(int)\n",
        "\n",
        "# Creating a dictionary with frequency of words\n",
        "for word in corpus_faked:\n",
        "    dic[word]+=1\n",
        "    \n",
        "# Sorting words by descending frequency\n",
        "top = sorted(dic.items(), key=lambda x: x[1], reverse=True)[:10]\n",
        "top"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B4xqFATubTzw"
      },
      "outputs": [],
      "source": [
        "twitter_mask = np.array(Image.open('/kaggle/input/masksforwordclouds/twitter_mask3.jpg'))\n",
        "\n",
        "wc = WordCloud(\n",
        "    background_color='white', \n",
        "    max_words=200, \n",
        "    mask=twitter_mask,\n",
        ")\n",
        "wc.generate(' '.join(text for text in df.loc[df['target'] == 0, 'cleaned_text']))\n",
        "plt.figure(figsize=(18,10))\n",
        "plt.title('Wordcloud for fake disasters', \n",
        "          fontdict={'size': 22,  'verticalalignment': 'bottom'})\n",
        "plt.imshow(wc)\n",
        "plt.axis(\"off\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "93oumaJObTzw"
      },
      "source": [
        "<h2>Model creation and testing</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tG0vorribTzw"
      },
      "outputs": [],
      "source": [
        "x = df['cleaned_text']\n",
        "y = df['target']\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "x_train, x_test, y_train, y_test = train_test_split(x, y)\n",
        "len(x_train), len(y_train), len(x_test), len(y_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dkdZoJYVbTzw"
      },
      "outputs": [],
      "source": [
        "pipe = Pipeline([\n",
        "    ('bow', CountVectorizer()), \n",
        "    ('tfid', TfidfTransformer()),  \n",
        "    ('model', xgb.XGBClassifier(\n",
        "        use_label_encoder=False,\n",
        "        eval_metric='auc',\n",
        "    ))\n",
        "])\n",
        "\n",
        "pipe.fit(x_train, y_train)\n",
        "yp_test = pipe.predict(x_test)\n",
        "yp_train = pipe.predict(x_train)\n",
        "\n",
        "print('Training accuracy: {}'.format(metrics.accuracy_score(y_train, yp_train)))\n",
        "print('Testing accuracy: {}'.format(metrics.accuracy_score(y_test, yp_test)))\n",
        "\n",
        "seaborn_conf(y_test, yp_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TercnODvbTzw"
      },
      "source": [
        "<h3>GloVE - LSTM</h2>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rh-1scsQbTzw"
      },
      "outputs": [],
      "source": [
        "training = df['cleaned_text'].values\n",
        "testing = df_test['cleaned_text'].values\n",
        "# Target labels\n",
        "labels = df['target'].values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EC42lAPWbTzw"
      },
      "outputs": [],
      "source": [
        "# Word tokenizer\n",
        "tk = Tokenizer()\n",
        "tk.fit_on_texts(training)\n",
        "\n",
        "vocab = len(tk.word_index)+1\n",
        "vocab"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Lf1gQo1mbTzx"
      },
      "outputs": [],
      "source": [
        "def metric_calculation(y_test, y_pred):\n",
        "    print(\"F1-score: \", f1_score(y_pred, y_test))\n",
        "    print(\"Precision: \", precision_score(y_pred, y_test))\n",
        "    print(\"Recall: \", recall_score(y_pred, y_test))\n",
        "    print(\"Acuracy: \", accuracy_score(y_pred, y_test))\n",
        "    print(\"-\"*50)\n",
        "    print(classification_report(y_pred, y_test))\n",
        "    \n",
        "def embeddings(corpus): \n",
        "    return tk.texts_to_sequences(corpus)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2IutjFRjbTzx"
      },
      "outputs": [],
      "source": [
        "len_train = np.max(df['cleaned_text'].apply(lambda x: len(x)))\n",
        "len_train"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_bkBioLcbTzx"
      },
      "outputs": [],
      "source": [
        "train_padded_sentences = pad_sequences(\n",
        "    embeddings(training), \n",
        "    len_train, \n",
        "    padding='post'\n",
        ")\n",
        "test_padded_sentences = pad_sequences(\n",
        "    embeddings(testing), \n",
        "    len_train,\n",
        "    padding='post'\n",
        ")\n",
        "\n",
        "train_padded_sentences"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5SIJE5ZkbTzx"
      },
      "outputs": [],
      "source": [
        "# As we've already created a GloVe dictionary in the SMS-dataset, we will start with the matrix\n",
        "\n",
        "embedding_matrix = np.zeros((vocab, embedding_dim))\n",
        "\n",
        "for word, index in tk.word_index.items():\n",
        "    embedding_vector = embedding_dict.get(word)\n",
        "    if embedding_vector is not None:\n",
        "        embedding_matrix[index] = embedding_vector\n",
        "        \n",
        "embedding_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iRNmJYFDbTzx"
      },
      "outputs": [],
      "source": [
        "x_train, x_test, y_train, y_test = train_test_split(train_padded_sentences, labels, test_size=0.20)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BcdIDlzXbTzx"
      },
      "outputs": [],
      "source": [
        "model = Sequential()\n",
        "model.add(Embedding(input_dim=embedding_matrix.shape[0], \n",
        "                   output_dim=embedding_matrix.shape[1],\n",
        "                   weights=[embedding_matrix],\n",
        "                   input_length=max_len\n",
        "                   )\n",
        "         )\n",
        "model.add(Bidirectional(LSTM(max_len, return_sequences=True, recurrent_dropout=0.15)))\n",
        "model.add(GlobalMaxPool1D())\n",
        "model.add(BatchNormalization())\n",
        "model.add(Dropout(0.3))\n",
        "model.add(Dense(max_len, activation='relu'))\n",
        "model.add(Dropout(0.3))\n",
        "model.add(Dense(max_len, activation='relu'))\n",
        "model.add(Dropout(0.3))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "\n",
        "\n",
        "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pgBblxl8bTzx"
      },
      "outputs": [],
      "source": [
        "# Callbacks\n",
        "checkpoint = ModelCheckpoint('model.h5', monitor = 'val_loss', verbose = 1, save_best_only = True)\n",
        "reduce_lr = ReduceLROnPlateau(monitor = 'val_loss', factor = 0.2, verbose = 1, patience = 5,                        min_lr = 0.001)\n",
        "\n",
        "history = model.fit(x_train, y_train, epochs = 7,batch_size = 32,validation_data = (x_test, y_test),verbose = 1,callbacks = [reduce_lr, checkpoint])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iNj_eU4lbTzx"
      },
      "outputs": [],
      "source": [
        "learning_curve(history, [['loss', 'val_loss'],['accuracy', 'val_accuracy']])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CeYSI2M0bTzx"
      },
      "outputs": [],
      "source": [
        "preds = model.predict_classes(x_test)\n",
        "metric_calculation(preds, y_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c0PVci8NbTzx"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}