{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "view-in-github",
    "colab_type": "text"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/neomatrix369/awesome-ai-ml-dl/blob/master/examples/better-nlp/notebooks/google-colab/better_nlp_summarisers.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iFyUOrS1fczL"
   },
   "source": [
    "# Better NLP\n",
    "\n",
    "This is a wrapper program/library that encapsulates a couple of NLP libraries that are popular among the AI and ML communities.\n",
    "\n",
    "Examples have been used to illustrate the usage as much as possible. Not all the APIs of the underlying libraries have been covered.\n",
    "\n",
    "The idea is to keep the API language as high-level as possible, so its easier to use and stays human-readable.\n",
    "\n",
    "Libraries / frameworks covered:\n",
    "\n",
    "- nltk [site](http://www.nltk.org/) | [docs](https://buildmedia.readthedocs.org/media/pdf/nltk/latest/nltk.pdf)\n",
    "- numpy [site](https://www.numpy.org/) | [docs](https://docs.scipy.org/doc/)\n",
    "- networkx [site](https://networkx.github.io/) | [docs](https://networkx.github.io/documentation/stable/index.html)\n",
    "\n",
    "See [https://github.com/neomatrix369/awesome-ai-ml-dl/blob/master/examples/better-nlp](https://github.com/neomatrix369/awesome-ai-ml-dl/blob/master/examples/better-nlp) for more details.\n",
    "\n",
    "### This notebook will demonstrate the below NLP features / functionalities, using the above mentioned libraries\n",
    "\n",
    "- Cosine similarity summarisation technique (extractive summarisation)\n",
    "- Vertex ranking algorithm summarisation technique\n",
    "- Build a simple text summarisation tool using NLTK\n",
    "- Summarisation 4 (TODO)\n",
    "- Summarisation 5 (TODO)\n",
    "\n",
    "_Summarisation can be defined as a task of producing a concise and fluent summary while preserving key information and overall meaning._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IMTAIhz53w8G"
   },
   "source": [
    "### Resources\n",
    "\n",
    "- [Understand Text Summarization and create your own summarizer in python](https://towardsdatascience.com/understand-text-summarization-and-create-your-own-summarizer-in-python-b26a9f09fc70) or [An Introduction to Text Summarization using the TextRank Algorithm (with Python implementation)](https://www.analyticsvidhya.com/blog/2018/11/introduction-text-summarization-textrank-python/)\n",
    "- [Beyond bag of words: Using PyTextRank to find Phrases and Summarize text](https://medium.com/@aneesha/beyond-bag-of-words-using-pytextrank-to-find-phrases-and-summarize-text-f736fa3773c5)\n",
    "- [Build a simple text summarisation tool using NLTK](https://medium.com/@wilamelima/build-a-simple-text-summarisation-tool-using-nltk-ff0984fedb4f)\n",
    "- [Summarise Text with TFIDF in Python 1/2](https://towardsdatascience.com/tfidf-for-piece-of-text-in-python-43feccaa74f8) and [Summarise Text with TFIDF in Python 2/2](https://medium.com/@shivangisareen/summarise-text-with-tfidf-in-python-bc7ca10d3284)\n",
    "- [How to Make a Text Summarizer - Intro to Deep Learning #10 by Siraj Raval](https://www.youtube.com/watch?v=ogrJaOIuBx4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Lre8GErufczN"
   },
   "source": [
    "#### Setup and installation ( optional )\n",
    "\n",
    "In case, this notebook is running in a local environment (Linux/MacOS) or _Google Colab_ environment and in case it does not have the necessary dependencies installed then please execute the steps in the next section.\n",
    "\n",
    "Otherwise, please SKIP to the **Install Spacy model ( NOT optional )** section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 5927.0
    },
    "colab_type": "code",
    "id": "QJuCUOMOfczO",
    "outputId": "018e9102-de25-4fbe-b5b3-919c58644f00"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "apt-utils is already the newest version (1.4.9).\n",
      "dpkg is already the newest version (1.18.25).\n",
      "dselect is already the newest version (1.18.25).\n",
      "0 upgraded, 0 newly installed, 0 to remove and 78 not upgraded.\n",
      "OSTYPE=linux-gnu\n",
      "Library source found\n",
      "Detected OS: linux\n",
      "Please check if you fulfill the requirements mentioned in the README file.\n",
      "Ign:1 http://deb.debian.org/debian stretch InRelease\n",
      "Hit:2 http://security.debian.org/debian-security stretch/updates InRelease\n",
      "Get:3 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]\n",
      "Hit:4 http://deb.debian.org/debian stretch Release\n",
      "Hit:5 https://deb.nodesource.com/node_8.x stretch InRelease\n",
      "Fetched 91.0 kB in 1s (85.5 kB/s)\n",
      "Reading package lists...\n",
      "Reading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "libswscale-dev is already the newest version (7:3.2.14-1~deb9u1).\n",
      "liblapack-dev is already the newest version (3.7.0-2).\n",
      "pkg-config is already the newest version (0.29-4+b1).\n",
      "wget is already the newest version (1.18-5+deb9u3).\n",
      "curl is already the newest version (7.52.1-5+deb9u10).\n",
      "0 upgraded, 0 newly installed, 0 to remove and 78 not upgraded.\n",
      "Reading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "vim is already the newest version (2:8.0.0197-4+deb9u3).\n",
      "zip is already the newest version (3.0-11+b1).\n",
      "0 upgraded, 0 newly installed, 0 to remove and 78 not upgraded.\n",
      "\n",
      "================================================================================\n",
      "================================================================================\n",
      "\n",
      "                              DEPRECATION WARNING                            \n",
      "\n",
      "  Node.js 8.x LTS Carbon is no longer actively supported!\n",
      "\n",
      "  You will not receive security or critical stability updates for this version.\n",
      "\n",
      "  You should migrate to a supported version of Node.js as soon as possible.\n",
      "  Use the installation script that corresponds to the version of Node.js you\n",
      "  wish to install. e.g.\n",
      "\n",
      "   * https://deb.nodesource.com/setup_10.x — Node.js 10 LTS \"Dubnium\" (recommended)\n",
      "   * https://deb.nodesource.com/setup_12.x — Node.js 12 LTS \"Erbium\"\n",
      "\n",
      "  Please see https://github.com/nodejs/Release for details about which\n",
      "  version may be appropriate for you.\n",
      "\n",
      "  The NodeSource Node.js distributions repository contains\n",
      "  information both about supported versions of Node.js and supported Linux\n",
      "  distributions. To learn more about usage, see the repository:\n",
      "    https://github.com/nodesource/distributions\n",
      "\n",
      "================================================================================\n",
      "================================================================================\n",
      "\n",
      "Continuing in 20 seconds ...\n",
      "\n",
      "\n",
      "## Installing the NodeSource Node.js 8.x LTS Carbon repo...\n",
      "\n",
      "\n",
      "## Populating apt-get cache...\n",
      "\n",
      "+ apt-get update\n",
      "Hit:1 http://security.debian.org/debian-security stretch/updates InRelease\n",
      "Ign:2 http://deb.debian.org/debian stretch InRelease\n",
      "Hit:3 http://deb.debian.org/debian stretch-updates InRelease\n",
      "Hit:4 http://deb.debian.org/debian stretch Release\n",
      "Hit:5 https://deb.nodesource.com/node_8.x stretch InRelease\n",
      "Reading package lists...\n",
      "\n",
      "## Confirming \"stretch\" is supported...\n",
      "\n",
      "+ curl -sLf -o /dev/null 'https://deb.nodesource.com/node_8.x/dists/stretch/Release'\n",
      "\n",
      "## Adding the NodeSource signing key to your keyring...\n",
      "\n",
      "+ curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -\n",
      "OK\n",
      "\n",
      "## Creating apt sources list file for the NodeSource Node.js 8.x LTS Carbon repo...\n",
      "\n",
      "+ echo 'deb https://deb.nodesource.com/node_8.x stretch main' > /etc/apt/sources.list.d/nodesource.list\n",
      "+ echo 'deb-src https://deb.nodesource.com/node_8.x stretch main' >> /etc/apt/sources.list.d/nodesource.list\n",
      "\n",
      "## Running `apt-get update` for you...\n",
      "\n",
      "+ apt-get update\n",
      "Hit:1 http://security.debian.org/debian-security stretch/updates InRelease\n",
      "Ign:2 http://deb.debian.org/debian stretch InRelease\n",
      "Hit:3 http://deb.debian.org/debian stretch-updates InRelease\n",
      "Hit:4 http://deb.debian.org/debian stretch Release\n",
      "Hit:5 https://deb.nodesource.com/node_8.x stretch InRelease\n",
      "Reading package lists...\n",
      "\n",
      "## Run `sudo apt-get install -y nodejs` to install Node.js 8.x LTS Carbon and npm\n",
      "## You may also need development tools to build native addons:\n",
      "     sudo apt-get install gcc g++ make\n",
      "## To install the Yarn package manager, run:\n",
      "     curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -\n",
      "     echo \"deb https://dl.yarnpkg.com/debian/ stable main\" | sudo tee /etc/apt/sources.list.d/yarn.list\n",
      "     sudo apt-get update && sudo apt-get install yarn\n",
      "\n",
      "\n",
      "Reading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "nodejs is already the newest version (8.17.0-1nodesource1).\n",
      "0 upgraded, 0 newly installed, 0 to remove and 78 not upgraded.\n",
      "Install components using python and pip\n",
      "/usr/bin/npm -> /usr/lib/node_modules/npm/bin/npm-cli.js\n",
      "/usr/bin/npx -> /usr/lib/node_modules/npm/bin/npx-cli.js\n",
      "+ npm@6.14.2\n",
      "updated 1 package in 21.251s\n",
      "6.14.2\n",
      "Python 3.7.3\n",
      "pip 19.0.3 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)\n",
      "Requirement already satisfied: spacy in /usr/local/lib/python3.7/site-packages (2.2.1)\n",
      "Requirement already satisfied: textacy in /usr/local/lib/python3.7/site-packages (0.9.1)\n",
      "Requirement already satisfied: pytextrank in /usr/local/lib/python3.7/site-packages (1.1.0)\n",
      "Requirement already satisfied: nltk in /usr/local/lib/python3.7/site-packages (3.4.5)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/site-packages (from spacy) (2.22.0)\n",
      "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/site-packages (from spacy) (0.4.1)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.7/site-packages (from spacy) (0.2.2)\n",
      "Requirement already satisfied: thinc<7.2.0,>=7.1.1 in /usr/local/lib/python3.7/site-packages (from spacy) (7.1.1)\n",
      "Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.7/site-packages (from spacy) (0.9.6)\n",
      "Requirement already satisfied: srsly<1.1.0,>=0.1.0 in /usr/local/lib/python3.7/site-packages (from spacy) (0.1.0)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from spacy) (3.0.2)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/site-packages (from spacy) (1.0.2)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/site-packages (from spacy) (2.0.2)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/site-packages (from spacy) (1.17.2)\n",
      "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.7/site-packages (from textacy) (1.3.1)\n",
      "Requirement already satisfied: pyemd>=0.5.0 in /usr/local/lib/python3.7/site-packages (from textacy) (0.5.1)\n",
      "Requirement already satisfied: pyphen>=0.9.4 in /usr/local/lib/python3.7/site-packages (from textacy) (0.9.5)\n",
      "Requirement already satisfied: tqdm>=4.19.6 in /usr/local/lib/python3.7/site-packages (from textacy) (4.36.1)\n",
      "Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/site-packages (from textacy) (2.3)\n",
      "Requirement already satisfied: cytoolz>=0.8.0 in /usr/local/lib/python3.7/site-packages (from textacy) (0.10.0)\n",
      "Requirement already satisfied: cachetools>=2.0.1 in /usr/local/lib/python3.7/site-packages (from textacy) (3.1.1)\n",
      "Requirement already satisfied: joblib>=0.13.0 in /usr/local/lib/python3.7/site-packages (from textacy) (0.14.0)\n",
      "Requirement already satisfied: scikit-learn>=0.19.0 in /usr/local/lib/python3.7/site-packages (from textacy) (0.21.3)\n",
      "Requirement already satisfied: jellyfish>=0.7.0 in /usr/local/lib/python3.7/site-packages (from textacy) (0.7.2)\n",
      "Requirement already satisfied: graphviz in /usr/local/lib/python3.7/site-packages (from pytextrank) (0.13)\n",
      "Requirement already satisfied: datasketch in /usr/local/lib/python3.7/site-packages (from pytextrank) (1.4.10)\n",
      "Requirement already satisfied: statistics in /usr/local/lib/python3.7/site-packages (from pytextrank) (1.0.3.5)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.7/site-packages (from nltk) (1.12.0)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2.8)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.0.4)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy) (1.25.6)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2019.9.11)\n",
      "Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.7/site-packages (from networkx>=2.0->textacy) (4.4.0)\n",
      "Requirement already satisfied: toolz>=0.8.0 in /usr/local/lib/python3.7/site-packages (from cytoolz>=0.8.0->textacy) (0.10.0)\n",
      "Requirement already satisfied: redis>=2.10.0 in /usr/local/lib/python3.7/site-packages (from datasketch->pytextrank) (3.3.8)\n",
      "Requirement already satisfied: docutils>=0.3 in /usr/local/lib/python3.7/site-packages (from statistics->pytextrank) (0.15.2)\n",
      "Requirement already satisfied: jupyterlab in /usr/local/lib/python3.7/site-packages (1.1.4)\n",
      "Requirement already satisfied: pandas in /usr/local/lib/python3.7/site-packages (0.25.1)\n",
      "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/site-packages (3.1.1)\n",
      "Requirement already satisfied: tornado!=6.0.0,!=6.0.1,!=6.0.2 in /usr/local/lib/python3.7/site-packages (from jupyterlab) (6.0.3)\n",
      "Requirement already satisfied: jinja2>=2.10 in /usr/local/lib/python3.7/site-packages (from jupyterlab) (2.10.3)\n",
      "Requirement already satisfied: jupyterlab-server~=1.0.0 in /usr/local/lib/python3.7/site-packages (from jupyterlab) (1.0.6)\n",
      "Requirement already satisfied: notebook>=4.3.1 in /usr/local/lib/python3.7/site-packages (from jupyterlab) (6.0.1)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/site-packages (from pandas) (2019.3)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/site-packages (from pandas) (1.17.2)\n",
      "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.7/site-packages (from pandas) (2.8.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/site-packages (from matplotlib) (1.1.0)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/site-packages (from matplotlib) (2.4.2)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/site-packages (from matplotlib) (0.10.0)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/site-packages (from jinja2>=2.10->jupyterlab) (1.1.1)\n",
      "Requirement already satisfied: jsonschema>=3.0.1 in /usr/local/lib/python3.7/site-packages (from jupyterlab-server~=1.0.0->jupyterlab) (3.1.1)\n",
      "Requirement already satisfied: json5 in /usr/local/lib/python3.7/site-packages (from jupyterlab-server~=1.0.0->jupyterlab) (0.8.5)\n",
      "Requirement already satisfied: prometheus-client in /usr/local/lib/python3.7/site-packages (from notebook>=4.3.1->jupyterlab) (0.7.1)\n",
      "Requirement already satisfied: jupyter-core>=4.4.0 in /usr/local/lib/python3.7/site-packages (from notebook>=4.3.1->jupyterlab) (4.6.0)\n",
      "Requirement already satisfied: nbconvert in /usr/local/lib/python3.7/site-packages (from notebook>=4.3.1->jupyterlab) (5.6.0)\n",
      "Requirement already satisfied: jupyter-client>=5.3.1 in /usr/local/lib/python3.7/site-packages (from notebook>=4.3.1->jupyterlab) (5.3.4)\n",
      "Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.7/site-packages (from notebook>=4.3.1->jupyterlab) (0.2.0)\n",
      "Requirement already satisfied: nbformat in /usr/local/lib/python3.7/site-packages (from notebook>=4.3.1->jupyterlab) (4.4.0)\n",
      "Requirement already satisfied: terminado>=0.8.1 in /usr/local/lib/python3.7/site-packages (from notebook>=4.3.1->jupyterlab) (0.8.2)\n",
      "Requirement already satisfied: ipykernel in /usr/local/lib/python3.7/site-packages (from notebook>=4.3.1->jupyterlab) (5.1.2)\n",
      "Requirement already satisfied: Send2Trash in /usr/local/lib/python3.7/site-packages (from notebook>=4.3.1->jupyterlab) (1.5.0)\n",
      "Requirement already satisfied: traitlets>=4.2.1 in /usr/local/lib/python3.7/site-packages (from notebook>=4.3.1->jupyterlab) (4.3.3)\n",
      "Requirement already satisfied: pyzmq>=17 in /usr/local/lib/python3.7/site-packages (from notebook>=4.3.1->jupyterlab) (18.1.0)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil>=2.6.1->pandas) (1.12.0)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib) (40.8.0)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.7/site-packages (from jsonschema>=3.0.1->jupyterlab-server~=1.0.0->jupyterlab) (19.2.0)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /usr/local/lib/python3.7/site-packages (from jsonschema>=3.0.1->jupyterlab-server~=1.0.0->jupyterlab) (0.15.4)\n",
      "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/site-packages (from jsonschema>=3.0.1->jupyterlab-server~=1.0.0->jupyterlab) (0.23)\n",
      "Requirement already satisfied: testpath in /usr/local/lib/python3.7/site-packages (from nbconvert->notebook>=4.3.1->jupyterlab) (0.4.2)\n",
      "Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.7/site-packages (from nbconvert->notebook>=4.3.1->jupyterlab) (0.3)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.7/site-packages (from nbconvert->notebook>=4.3.1->jupyterlab) (0.8.4)\n",
      "Requirement already satisfied: defusedxml in /usr/local/lib/python3.7/site-packages (from nbconvert->notebook>=4.3.1->jupyterlab) (0.6.0)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.7/site-packages (from nbconvert->notebook>=4.3.1->jupyterlab) (1.4.2)\n",
      "Requirement already satisfied: bleach in /usr/local/lib/python3.7/site-packages (from nbconvert->notebook>=4.3.1->jupyterlab) (3.1.0)\n",
      "Requirement already satisfied: pygments in /usr/local/lib/python3.7/site-packages (from nbconvert->notebook>=4.3.1->jupyterlab) (2.4.2)\n",
      "Requirement already satisfied: ptyprocess; os_name != \"nt\" in /usr/local/lib/python3.7/site-packages (from terminado>=0.8.1->notebook>=4.3.1->jupyterlab) (0.6.0)\n",
      "Requirement already satisfied: ipython>=5.0.0 in /usr/local/lib/python3.7/site-packages (from ipykernel->notebook>=4.3.1->jupyterlab) (7.8.0)\n",
      "Requirement already satisfied: decorator in /usr/local/lib/python3.7/site-packages (from traitlets>=4.2.1->notebook>=4.3.1->jupyterlab) (4.4.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/site-packages (from importlib-metadata->jsonschema>=3.0.1->jupyterlab-server~=1.0.0->jupyterlab) (0.6.0)\n",
      "Requirement already satisfied: webencodings in /usr/local/lib/python3.7/site-packages (from bleach->nbconvert->notebook>=4.3.1->jupyterlab) (0.5.1)\n",
      "Requirement already satisfied: backcall in /usr/local/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel->notebook>=4.3.1->jupyterlab) (0.1.0)\n",
      "Requirement already satisfied: pexpect; sys_platform != \"win32\" in /usr/local/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel->notebook>=4.3.1->jupyterlab) (4.7.0)\n",
      "Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel->notebook>=4.3.1->jupyterlab) (0.7.5)\n",
      "Requirement already satisfied: jedi>=0.10 in /usr/local/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel->notebook>=4.3.1->jupyterlab) (0.15.1)\n",
      "Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in /usr/local/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel->notebook>=4.3.1->jupyterlab) (2.0.10)\n",
      "Requirement already satisfied: more-itertools in /usr/local/lib/python3.7/site-packages (from zipp>=0.5->importlib-metadata->jsonschema>=3.0.1->jupyterlab-server~=1.0.0->jupyterlab) (7.2.0)\n",
      "Requirement already satisfied: parso>=0.5.0 in /usr/local/lib/python3.7/site-packages (from jedi>=0.10->ipython>=5.0.0->ipykernel->notebook>=4.3.1->jupyterlab) (0.5.1)\n",
      "Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/site-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython>=5.0.0->ipykernel->notebook>=4.3.1->jupyterlab) (0.1.7)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning: apt-key output should not be parsed (stdout is not a terminal)\n",
      "You are using pip version 19.0.3, however version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "You are using pip version 19.0.3, however version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 24.7 ms, sys: 18.5 ms, total: 43.2 ms\n",
      "Wall time: 1min 5s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "%%bash\n",
    "\n",
    "apt-get install apt-utils dselect dpkg\n",
    "\n",
    "echo \"OSTYPE=$OSTYPE\"\n",
    "if [[ \"$OSTYPE\" == \"cygwin\" ]] || [[ \"$OSTYPE\" == \"msys\" ]] ; then\n",
    "    echo \"Windows or Windows-like environment detected, script not tested, and may not work.\"\n",
    "    echo \"Try installing the components mention in the install-[ostype].sh scripts manually.\"\n",
    "    echo \"Or try running under CGYWIN or git-bash.\"\n",
    "    echo \"If successfully installed, please contribute back with the solution via a pull request, to https://github.com/neomatrix369/awesome-ai-ml-dl/\"\n",
    "    echo \"Please give the file a good name, i.e. install-windows.sh or install-windows.bat depending on what kind of script you end up writing\"\n",
    "    exit 0\n",
    "elif [[ \"$OSTYPE\" == \"linux-gnu\" ]] || [[ \"$OSTYPE\" == \"linux\" ]]; then\n",
    "    TARGET_OS=\"linux\"\n",
    "else\n",
    "    TARGET_OS=\"macos\"\n",
    "fi\n",
    "\n",
    "if [[ -e ../../library/org/neomatrix369 ]]; then\n",
    "  echo \"Library source found\"\n",
    "  \n",
    "  cd ../../build\n",
    "  \n",
    "  echo \"Detected OS: ${TARGET_OS}\"\n",
    "  ./install-${TARGET_OS}.sh || true\n",
    "else\n",
    "  if [[ -e awesome-ai-ml-dl/examples/better-nlp/library ]]; then\n",
    "     echo \"Library source found\"\n",
    "  else\n",
    "     git clone \"https://github.com/neomatrix369/awesome-ai-ml-dl\"\n",
    "  fi\n",
    "\n",
    "  echo \"Library source exists\"\n",
    "  cd awesome-ai-ml-dl/examples/better-nlp/build\n",
    "\n",
    "  echo \"Detected OS: ${TARGET_OS}\"\n",
    "  ./install-${TARGET_OS}.sh || true \n",
    "fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vPLdwvt63w8R"
   },
   "source": [
    "#### Install Spacy model ( NOT optional )\n",
    "\n",
    "Install the large English language model for spaCy - will be needed for the examples in this notebooks.\n",
    "\n",
    "**Note:** from observation it appears that spaCy model should be installed towards the end of the installation process, it avoid errors when running programs using the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-dJrJ54a3w8S",
    "outputId": "3da4300b-89a8-43e3-e989-fa7859ecdfc8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: en_core_web_lg==2.2.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.2.0/en_core_web_lg-2.2.0.tar.gz#egg=en_core_web_lg==2.2.0 in /usr/local/lib/python3.7/site-packages (2.2.0)\n",
      "Requirement already satisfied: spacy>=2.2.0 in /usr/local/lib/python3.7/site-packages (from en_core_web_lg==2.2.0) (2.2.1)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from spacy>=2.2.0->en_core_web_lg==2.2.0) (3.0.2)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.7/site-packages (from spacy>=2.2.0->en_core_web_lg==2.2.0) (0.2.2)\n",
      "Requirement already satisfied: thinc<7.2.0,>=7.1.1 in /usr/local/lib/python3.7/site-packages (from spacy>=2.2.0->en_core_web_lg==2.2.0) (7.1.1)\n",
      "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/site-packages (from spacy>=2.2.0->en_core_web_lg==2.2.0) (0.4.1)\n",
      "Requirement already satisfied: srsly<1.1.0,>=0.1.0 in /usr/local/lib/python3.7/site-packages (from spacy>=2.2.0->en_core_web_lg==2.2.0) (0.1.0)\n",
      "Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.7/site-packages (from spacy>=2.2.0->en_core_web_lg==2.2.0) (0.9.6)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/site-packages (from spacy>=2.2.0->en_core_web_lg==2.2.0) (1.17.2)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/site-packages (from spacy>=2.2.0->en_core_web_lg==2.2.0) (2.22.0)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/site-packages (from spacy>=2.2.0->en_core_web_lg==2.2.0) (1.0.2)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/site-packages (from spacy>=2.2.0->en_core_web_lg==2.2.0) (2.0.2)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /usr/local/lib/python3.7/site-packages (from thinc<7.2.0,>=7.1.1->spacy>=2.2.0->en_core_web_lg==2.2.0) (4.36.1)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.0->en_core_web_lg==2.2.0) (3.0.4)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.0->en_core_web_lg==2.2.0) (1.25.6)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.0->en_core_web_lg==2.2.0) (2019.9.11)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.0->en_core_web_lg==2.2.0) (2.8)\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the model via spacy.load('en_core_web_lg')\n",
      "\n",
      "\u001b[38;5;1m✘ Link 'en' already exists\u001b[0m\n",
      "To overwrite an existing link, use the --force flag\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 19.0.3, however version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 9.32 ms, sys: 9.09 ms, total: 18.4 ms\n",
      "Wall time: 6.84 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "%%bash\n",
    "\n",
    "python -m spacy download en_core_web_lg\n",
    "python -m spacy link en_core_web_lg en || true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kwXgEdM8oeUv"
   },
   "source": [
    "## Examples of various summarisation methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AX1pZlKofczb"
   },
   "source": [
    "### 1. Cosine similarity summarisation technique (extractive summarisation)\n",
    "\n",
    "**Abstractive Summarization:** Abstractive methods select words based on semantic understanding, even those words did not appear in the source documents. It aims at producing important material in a new way. They interpret and examine the text using advanced natural language techniques in order to generate a new shorter text that conveys the most critical information from the original text.\n",
    "\n",
    "**Flow:** Input document → understand context → semantics → create own summary\n",
    "\n",
    "**Extractive Summarization:** Extractive methods attempt to summarize articles by selecting a subset of words that retain the most important points.\n",
    "\n",
    "**Flow:** Input document → sentences similarity → weight sentences → select sentences with higher rank\n",
    "\n",
    "**Cosine similarity** is a measure of similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them. Its measures cosine of the angle between vectors. Angle will be 0 if sentences are similar and tend towards 90 as they begin to differ.\n",
    "\n",
    "Inspired by Praveen Dubey the author of https://towardsdatascience.com/understand-text-summarization-and-create-your-own-summarizer-in-python-b26a9f09fc70\n",
    "\n",
    "or see [Understand Text Summarization and create your own summarizer in python](https://www.analyticsvidhya.com/blog/2018/11/introduction-text-summarization-textrank-python/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Yjan7P5_3w8Z"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This version of Python is 64 bits.\n",
      "This version of Python is 64 bits.\n",
      "This version of Python is 64 bits.\n",
      "This version of Python is 64 bits.\n",
      "This version of Python is 64 bits.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '../../library')\n",
    "sys.path.insert(0, './awesome-ai-ml-dl/examples/better-nlp/library')\n",
    "\n",
    "from org.neomatrix369.better_nlp import BetterNLP\n",
    "\n",
    "import pprint\n",
    "pp = pprint.PrettyPrinter(indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zpjkLRHe3w8c"
   },
   "outputs": [],
   "source": [
    "betterNLP = BetterNLP() ### do not re-run this unless you wish to re-initialise the object\n",
    "generic_text=\"\"\"In an attempt to build an AI-ready workforce, SmartSoft Corp. announced Smart Colab Program which has been launched to empower the next generation of students with AI-ready skills. Envisioned as a three-year collaborative program, Smart Colab Program will support around 100 institutions with AI infrastructure, course content and curriculum, developer support, development tools and give students access to cloud and AI services. As part of the program, the Palo Alto giant which wants to expand its reach and is planning to build a strong developer ecosystem in India with the program will set up the core AI infrastructure and IoT Hub for the selected campuses. The company will provide AI development tools and AI services such as SmartSoft Corp. Cognitive Services, Bot Services and Machine Learning Services. According to Mark Smith, Country AI Manager, SmartSoft Corp. India, said, \"With AI being the defining technology of our time, it is transforming lives and industry and the jobs of tomorrow will require a different skillset. This will require more collaborations and training and working with AI. That’s why it has become more critical than ever for educational institutions to integrate new cloud and AI technologies. The program is an attempt to ramp up the institutional set-up and build capabilities among the educators to educate the workforce of tomorrow.\" The program aims to build up the cognitive skills and in-depth understanding of developing intelligent cloud connected solutions for applications across industry. Earlier in April this year, the company announced SmartSoft Corp. Advanced Program In AI as a learning track open to the public. The program was developed to provide job ready skills to programmers who wanted to hone their skills in AI and data science with a series of online courses which featured hands-on labs and expert instructors as well. This program also included developer-focused AI school that provided a bunch of assets to help build AI skills.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-vkYIGs13w8f",
    "outputId": "083398af-ca33-4c1c-f892-e1567de17931"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "summarisation_processing_time_in_secs= 0.1727583408355713\n",
      "('summarised_text=The company will provide AI development tools and AI '\n",
      " 'services such as SmartSoft Corp. Envisioned as a three-year collaborative '\n",
      " 'program, Smart Colab Program will support around 100 institutions with AI '\n",
      " 'infrastructure, course content and curriculum, developer support, '\n",
      " 'development tools and give students access to cloud and AI services. '\n",
      " 'According to Mark Smith, Country AI Manager, SmartSoft Corp. Advanced '\n",
      " 'Program In AI as a learning track open to the public. This will require more '\n",
      " 'collaborations and training and working with AI')\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n"
     ]
    }
   ],
   "source": [
    "summarised_result = betterNLP.summarise(generic_text)\n",
    "\n",
    "print(\"~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\")\n",
    "print(\"summarisation_processing_time_in_secs=\",summarised_result['summarisation_processing_time_in_secs'])\n",
    "pp.pprint(\"summarised_text=\" + summarised_result['summarised_text'])\n",
    "print(\"~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ranked_sentences=\n",
      "[   (   0.13833101031465372,\n",
      "        [   'The',\n",
      "            'company',\n",
      "            'will',\n",
      "            'provide',\n",
      "            'AI',\n",
      "            'development',\n",
      "            'tools',\n",
      "            'and',\n",
      "            'AI',\n",
      "            'services',\n",
      "            'such',\n",
      "            'as',\n",
      "            'SmartSoft',\n",
      "            'Corp']),\n",
      "    (   0.10383640827971251,\n",
      "        [   'Envisioned',\n",
      "            'as',\n",
      "            'a',\n",
      "            'three-year',\n",
      "            'collaborative',\n",
      "            'program,',\n",
      "            'Smart',\n",
      "            'Colab',\n",
      "            'Program',\n",
      "            'will',\n",
      "            'support',\n",
      "            'around',\n",
      "            '100',\n",
      "            'institutions',\n",
      "            'with',\n",
      "            'AI',\n",
      "            'infrastructure,',\n",
      "            'course',\n",
      "            'content',\n",
      "            'and',\n",
      "            'curriculum,',\n",
      "            'developer',\n",
      "            'support,',\n",
      "            'development',\n",
      "            'tools',\n",
      "            'and',\n",
      "            'give',\n",
      "            'students',\n",
      "            'access',\n",
      "            'to',\n",
      "            'cloud',\n",
      "            'and',\n",
      "            'AI',\n",
      "            'services']),\n",
      "    (   0.08993180993446229,\n",
      "        [   'According',\n",
      "            'to',\n",
      "            'Mark',\n",
      "            'Smith,',\n",
      "            'Country',\n",
      "            'AI',\n",
      "            'Manager,',\n",
      "            'SmartSoft',\n",
      "            'Corp']),\n",
      "    (   0.08705232496568506,\n",
      "        [   'Advanced',\n",
      "            'Program',\n",
      "            'In',\n",
      "            'AI',\n",
      "            'as',\n",
      "            'a',\n",
      "            'learning',\n",
      "            'track',\n",
      "            'open',\n",
      "            'to',\n",
      "            'the',\n",
      "            'public']),\n",
      "    (   0.07027286099160632,\n",
      "        [   'This',\n",
      "            'will',\n",
      "            'require',\n",
      "            'more',\n",
      "            'collaborations',\n",
      "            'and',\n",
      "            'training',\n",
      "            'and',\n",
      "            'working',\n",
      "            'with',\n",
      "            'AI']),\n",
      "    (   0.06876231604969901,\n",
      "        [   'In',\n",
      "            'an',\n",
      "            'attempt',\n",
      "            'to',\n",
      "            'build',\n",
      "            'an',\n",
      "            'AI-ready',\n",
      "            'workforce,',\n",
      "            'SmartSoft',\n",
      "            'Corp']),\n",
      "    (   0.06333630121561608,\n",
      "        [   'The',\n",
      "            'program',\n",
      "            'was',\n",
      "            'developed',\n",
      "            'to',\n",
      "            'provide',\n",
      "            'job',\n",
      "            'ready',\n",
      "            'skills',\n",
      "            'to',\n",
      "            'programmers',\n",
      "            'who',\n",
      "            'wanted',\n",
      "            'to',\n",
      "            'hone',\n",
      "            'their',\n",
      "            'skills',\n",
      "            'in',\n",
      "            'AI',\n",
      "            'and',\n",
      "            'data',\n",
      "            'science',\n",
      "            'with',\n",
      "            'a',\n",
      "            'series',\n",
      "            'of',\n",
      "            'online',\n",
      "            'courses',\n",
      "            'which',\n",
      "            'featured',\n",
      "            'hands-on',\n",
      "            'labs',\n",
      "            'and',\n",
      "            'expert',\n",
      "            'instructors',\n",
      "            'as',\n",
      "            'well']),\n",
      "    (   0.06301075479451754,\n",
      "        [   'As',\n",
      "            'part',\n",
      "            'of',\n",
      "            'the',\n",
      "            'program,',\n",
      "            'the',\n",
      "            'Palo',\n",
      "            'Alto',\n",
      "            'giant',\n",
      "            'which',\n",
      "            'wants',\n",
      "            'to',\n",
      "            'expand',\n",
      "            'its',\n",
      "            'reach',\n",
      "            'and',\n",
      "            'is',\n",
      "            'planning',\n",
      "            'to',\n",
      "            'build',\n",
      "            'a',\n",
      "            'strong',\n",
      "            'developer',\n",
      "            'ecosystem',\n",
      "            'in',\n",
      "            'India',\n",
      "            'with',\n",
      "            'the',\n",
      "            'program',\n",
      "            'will',\n",
      "            'set',\n",
      "            'up',\n",
      "            'the',\n",
      "            'core',\n",
      "            'AI',\n",
      "            'infrastructure',\n",
      "            'and',\n",
      "            'IoT',\n",
      "            'Hub',\n",
      "            'for',\n",
      "            'the',\n",
      "            'selected',\n",
      "            'campuses']),\n",
      "    (   0.061233356253838966,\n",
      "        [   'The',\n",
      "            'program',\n",
      "            'is',\n",
      "            'an',\n",
      "            'attempt',\n",
      "            'to',\n",
      "            'ramp',\n",
      "            'up',\n",
      "            'the',\n",
      "            'institutional',\n",
      "            'set-up',\n",
      "            'and',\n",
      "            'build',\n",
      "            'capabilities',\n",
      "            'among',\n",
      "            'the',\n",
      "            'educators',\n",
      "            'to',\n",
      "            'educate',\n",
      "            'the',\n",
      "            'workforce',\n",
      "            'of',\n",
      "            'tomorrow.\"',\n",
      "            'The',\n",
      "            'program',\n",
      "            'aims',\n",
      "            'to',\n",
      "            'build',\n",
      "            'up',\n",
      "            'the',\n",
      "            'cognitive',\n",
      "            'skills',\n",
      "            'and',\n",
      "            'in-depth',\n",
      "            'understanding',\n",
      "            'of',\n",
      "            'developing',\n",
      "            'intelligent',\n",
      "            'cloud',\n",
      "            'connected',\n",
      "            'solutions',\n",
      "            'for',\n",
      "            'applications',\n",
      "            'across',\n",
      "            'industry']),\n",
      "    (   0.05663981471429329,\n",
      "        [   'Earlier',\n",
      "            'in',\n",
      "            'April',\n",
      "            'this',\n",
      "            'year,',\n",
      "            'the',\n",
      "            'company',\n",
      "            'announced',\n",
      "            'SmartSoft',\n",
      "            'Corp']),\n",
      "    (   0.056569479333572974,\n",
      "        [   'That’s',\n",
      "            'why',\n",
      "            'it',\n",
      "            'has',\n",
      "            'become',\n",
      "            'more',\n",
      "            'critical',\n",
      "            'than',\n",
      "            'ever',\n",
      "            'for',\n",
      "            'educational',\n",
      "            'institutions',\n",
      "            'to',\n",
      "            'integrate',\n",
      "            'new',\n",
      "            'cloud',\n",
      "            'and',\n",
      "            'AI',\n",
      "            'technologies']),\n",
      "    (   0.05626875135705457,\n",
      "        [   'announced',\n",
      "            'Smart',\n",
      "            'Colab',\n",
      "            'Program',\n",
      "            'which',\n",
      "            'has',\n",
      "            'been',\n",
      "            'launched',\n",
      "            'to',\n",
      "            'empower',\n",
      "            'the',\n",
      "            'next',\n",
      "            'generation',\n",
      "            'of',\n",
      "            'students',\n",
      "            'with',\n",
      "            'AI-ready',\n",
      "            'skills']),\n",
      "    (   0.051380215024656437,\n",
      "        [   'India,',\n",
      "            'said,',\n",
      "            '\"With',\n",
      "            'AI',\n",
      "            'being',\n",
      "            'the',\n",
      "            'defining',\n",
      "            'technology',\n",
      "            'of',\n",
      "            'our',\n",
      "            'time,',\n",
      "            'it',\n",
      "            'is',\n",
      "            'transforming',\n",
      "            'lives',\n",
      "            'and',\n",
      "            'industry',\n",
      "            'and',\n",
      "            'the',\n",
      "            'jobs',\n",
      "            'of',\n",
      "            'tomorrow',\n",
      "            'will',\n",
      "            'require',\n",
      "            'a',\n",
      "            'different',\n",
      "            'skillset']),\n",
      "    (   0.03337459677063151,\n",
      "        [   'Cognitive',\n",
      "            'Services,',\n",
      "            'Bot',\n",
      "            'Services',\n",
      "            'and',\n",
      "            'Machine',\n",
      "            'Learning',\n",
      "            'Services'])]\n"
     ]
    }
   ],
   "source": [
    "print(\"ranked_sentences=\") \n",
    "pp.pprint(summarised_result['ranked_sentences'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "T7A3p05vfcz0"
   },
   "source": [
    "### 2. Vertex ranking algorithm summarisation technique\n",
    "\n",
    "Using PyTextRank to find Phrases and Summarize text: Multi-word Phrase Extraction and Sentence Extraction for Summarization\n",
    "\n",
    "Inspired by the author of https://medium.com/@aneesha/beyond-bag-of-words-using-pytextrank-to-find-phrases-and-summarize-text-f736fa3773c5 \n",
    "(Notebook: https://github.com/DerwenAI/pytextrank/blob/master/example.ipynb)\n",
    "\n",
    "Another resource to take a look at: https://www.analyticsvidhya.com/blog/2018/11/introduction-text-summarization-textrank-python/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "R0by6FnE3w8n"
   },
   "outputs": [],
   "source": [
    "betterNLP = BetterNLP() ### do not re-run this unless you wish to re-initialise the object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "w44pghUL3w8r",
    "outputId": "885ba416-64b3-48ef-b27c-9b66f2ca069f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sentence_text:  ['In', 'an', 'attempt', 'to', 'build', 'an', 'AI-ready', 'workforce', ',', 'SmartSoft', 'Corp.', 'announced', 'Smart', 'Colab', 'Program', 'which', 'has', 'been', 'launched', 'to', 'empower', 'the', 'next', 'generation', 'of', 'students', 'with', 'AI-ready', 'skills', '.']\n",
      "make sentence(sentence_text): In an attempt to build an AI-ready workforce, SmartSoft Corp. announced Smart Colab Program which has been launched to empower the next generation of students with AI-ready skills.\n",
      "sentence_text:  ['Envisioned', 'as', 'a', 'three-year', 'collaborative', 'program', ',', 'Smart', 'Colab', 'Program', 'will', 'support', 'around', '100', 'institutions', 'with', 'AI', 'infrastructure', ',', 'course', 'content', 'and', 'curriculum', ',', 'developer', 'support', ',', 'development', 'tools', 'and', 'give', 'students', 'access', 'to', 'cloud', 'and', 'AI', 'services', '.']\n",
      "make sentence(sentence_text): Envisioned as a three-year collaborative program, Smart Colab Program will support around 100 institutions with AI infrastructure, course content and curriculum, developer support, development tools and give students access to cloud and AI services.\n",
      "sentence_text:  ['The', 'company', 'will', 'provide', 'AI', 'development', 'tools', 'and', 'AI', 'services', 'such', 'as', 'SmartSoft', 'Corp.', 'Cognitive', 'Services', ',', 'Bot', 'Services', 'and', 'Machine', 'Learning', 'Services', '.']\n",
      "make sentence(sentence_text): The company will provide AI development tools and AI services such as SmartSoft Corp. Cognitive Services, Bot Services and Machine Learning Services.\n",
      "sentence_text:  ['SmartSoft', 'Corp.', 'Advanced', 'Program']\n",
      "make sentence(sentence_text): SmartSoft Corp. Advanced Program\n",
      "sentence_text:  ['This', 'program', 'also', 'included', 'developer-focused', 'AI', 'school', 'that', 'provided', 'a', 'bunch', 'of', 'assets', 'to', 'help', 'build', 'AI', 'skills', '.']\n",
      "make sentence(sentence_text): This program also included developer-focused AI school that provided a bunch of assets to help build AI skills.\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "summarisation_processing_time_in_secs= 40.71376895904541\n",
      "summarised_text= In an attempt to build an AI-ready workforce, SmartSoft Corp. announced Smart Colab Program which has been launched to empower the next generation of students with AI-ready skills. Envisioned as a three-year collaborative program, Smart Colab Program will support around 100 institutions with AI infrastructure, course content and curriculum, developer support, development tools and give students access to cloud and AI services. The company will provide AI development tools and AI services such as SmartSoft Corp. Cognitive Services, Bot Services and Machine Learning Services. SmartSoft Corp. Advanced Program This program also included developer-focused AI school that provided a bunch of assets to help build AI skills.\n",
      "token_ranks= {'attempt': 0.003915874149629682, 'build': 0.019017539757316935, 'AI-ready': 0.006608783584891741, 'workforce': 0.01058504555823971, 'SmartSoft': 0.018395189913830868, 'Corp.': 0.015652730434270696, 'announce': 0.005405230704102744, 'Smart': 0.007034708638636284, 'Colab': 0.007987227338711586, 'Program': 0.015962202795504806, 'have': 0.008901082294876322, 'be': 0.024955591036820036, 'launch': 0.0052692023724527385, 'empower': 0.007856728598311317, 'next': 0.004788402700870097, 'generation': 0.00682360539543684, 'student': 0.014770590241008671, 'skill': 0.032296964177813754, 'collaborative': 0.0014490520109942432, 'program': 0.007598643299902847, 'support': 0.012863294561197678, 'institution': 0.005709753654603553, 'AI': 0.047259774630369955, 'infrastructure': 0.008013628715638488, 'course': 0.009947751871804439, 'content': 0.00526607927664222, 'curriculum': 0.008039498435587744, 'developer': 0.008361521373588013, 'development': 0.010362691422438241, 'tool': 0.012990212722110154, 'give': 0.0064119037278227336, 'access': 0.007313287730281746, 'cloud': 0.01744669513681032, 'service': 0.010698856270961656, 'part': 0.0014490520109942432, 'Palo': 0.0019873139744724924, 'Alto': 0.0022936798893682425, 'giant': 0.003268510984821586, 'want': 0.017245727495979697, 'expand': 0.008779081720203951, 'reach': 0.008911886798289272, 'plan': 0.007165519023245537, 'strong': 0.0034697957468247543, 'ecosystem': 0.004700721628507262, 'India': 0.010121779976887492, 'set': 0.0019873139744724924, 'core': 0.003138307767742242, 'IoT': 0.005160737890418162, 'Hub': 0.007538937488812472, 'select': 0.007857533456518252, 'campus': 0.008128369001239779, 'company': 0.003727725784866856, 'provide': 0.01221355636116303, 'such': 0.008005108700843347, 'Cognitive': 0.004349027586382795, 'Services': 0.039586867892486624, 'Bot': 0.011703289770062462, 'Machine': 0.014825555160208231, 'Learning': 0.01615085429148529, 'accord': 0.0014490520109942432, 'Mark': 0.002064905973441604, 'Smith': 0.002942501398133437, 'Country': 0.0035772262700332876, 'Manager': 0.004978082737857278, 'say': 0.00722950353455673, 'define': 0.005386301447130493, 'technology': 0.012436786031099994, 'time': 0.012021173133637049, 'transform': 0.0033776121861514327, 'life': 0.00481318815530263, 'industry': 0.01753084695711191, 'job': 0.01926656662949509, 'tomorrow': 0.009907696634021149, 'require': 0.009871337639574401, 'different': 0.003546903600152163, 'skillset': 0.006561962428920655, 'more': 0.003546903600152163, 'collaboration': 0.005054433014536409, 'training': 0.0072531333639402505, 'work': 0.007614614751164673, 's': 0.0014490520109942432, 'become': 0.0033406421972955493, 'critical': 0.0061803145960477625, 'educational': 0.0014490520109942432, 'integrate': 0.0032782959276684035, 'new': 0.004055762246221477, 'ramp': 0.0031134018960720794, 'institutional': 0.004095562951079425, 'set-up': 0.0031897365977170316, 'capability': 0.004825475861787488, 'educator': 0.005550895005808473, 'educate': 0.006167531228934787, 'aim': 0.0019873139744724924, 'cognitive': 0.0034697957468247543, 'understanding': 0.010600707837529604, 'develop': 0.008421605975523671, 'intelligent': 0.007386577984640337, 'connect': 0.0074456895015524785, 'solution': 0.00917857226202877, 'application': 0.012416186096653131, 'April': 0.0014490520109942432, 'year': 0.002680759935888964, 'Advanced': 0.004349027586382795, 'learning': 0.0034577412719733035, 'track': 0.0029186719380210487, 'open': 0.005399637895286972, 'public': 0.006038938945932119, 'ready': 0.00839160842650517, 'programmer': 0.014167478419304085, 'hone': 0.008779081720203951, 'data': 0.0034577412719733035, 'science': 0.006396981126026914, 'series': 0.006886807929790258, 'online': 0.004376119134506868, 'feature': 0.005423029516071013, 'hand-on': 0.005867990013525933, 'lab': 0.006247988208115002, 'expert': 0.006598610919915678, 'instructor': 0.009713691056877237, 'include': 0.0019873139744724924, 'developer-focus': 0.0025503992267218254, 'school': 0.005104772491267291, 'bunch': 0.0029322223893143142, 'asset': 0.003941554290114426, 'help': 0.0031242779837859945}\n",
      "key_phrases= ai, smartsoft corp. cognitive services, ai-ready skills, core ai infrastructure, services, bot services and machine learning services, job ready skills, smartsoft corp., new cloud, cognitive skills, smart colab program, ai services, students\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/site-packages/networkx/drawing/nx_pylab.py:579: MatplotlibDeprecationWarning: \n",
      "The iterable function was deprecated in Matplotlib 3.1 and will be removed in 3.3. Use np.iterable instead.\n",
      "  if not cb.iterable(width):\n",
      "/usr/local/lib/python3.7/site-packages/networkx/drawing/nx_pylab.py:676: MatplotlibDeprecationWarning: \n",
      "The iterable function was deprecated in Matplotlib 3.1 and will be removed in 3.3. Use np.iterable instead.\n",
      "  if cb.iterable(node_size):  # many node sizes\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 72660 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "source_file='source.json'\n",
    "source_json_content='{\"id\":\"777\", \"text\":\"In an attempt to build an AI-ready workforce, SmartSoft Corp. announced Smart Colab Program which has been launched to empower the next generation of students with AI-ready skills. Envisioned as a three-year collaborative program, Smart Colab Program will support around 100 institutions with AI infrastructure, course content and curriculum, developer support, development tools and give students access to cloud and AI services. As part of the program, the Palo Alto giant which wants to expand its reach and is planning to build a strong developer ecosystem in India with the program will set up the core AI infrastructure and IoT Hub for the selected campuses. The company will provide AI development tools and AI services such as SmartSoft Corp. Cognitive Services, Bot Services and Machine Learning Services. According to Mark Smith, Country AI Manager, SmartSoft Corp. India, said, ''With AI being the defining technology of our time, it is transforming lives and industry and the jobs of tomorrow will require a different skillset. This will require more collaborations and training and working with AI. That''s why it has become more critical than ever for educational institutions to integrate new cloud and AI technologies. The program is an attempt to ramp up the institutional set-up and build capabilities among the educators to educate the workforce of tomorrow.'' The program aims to build up the cognitive skills and in-depth understanding of developing intelligent cloud connected solutions for applications across industry. Earlier in April this year, the company announced SmartSoft Corp. Advanced Program In AI as a learning track open to the public. The program was developed to provide job ready skills to programmers who wanted to hone their skills in AI and data science with a series of online courses which featured hands-on labs and expert instructors as well. This program also included developer-focused AI school that provided a bunch of assets to help build AI skills.\"}'\n",
    "f = open(source_file, 'w')\n",
    "f.write(\"%s\" % source_json_content)\n",
    "f.close()\n",
    "\n",
    "summarised_result = betterNLP.summarise(source_file, method=\"pytextrank\")\n",
    "\n",
    "print(\"~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\")\n",
    "print(\"summarisation_processing_time_in_secs=\",summarised_result['summarisation_processing_time_in_secs'])\n",
    "print(\"summarised_text=\",summarised_result['summarised_text'])\n",
    "print(\"token_ranks=\",summarised_result['token_ranks'])\n",
    "print(\"key_phrases=\",summarised_result['key_phrases'])\n",
    "print(\"~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\")\n",
    "betterNLP.show_graph(summarised_result[\"graph\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s2kyTI6bfcz7"
   },
   "source": [
    "### 3. Build a simple text summarisation tool using NLTK\n",
    "\n",
    "Inspired by Wilame Lima Vallantin, the author of [Build a simple text summarisation tool using NLTK](https://medium.com/@wilamelima/build-a-simple-text-summarisation-tool-using-nltk-ff0984fedb4f).\n",
    "\n",
    "We have to break the text into sentences and tokens, remove stop-words. Tokenise words, calculate word frequency to determine if a word is important on the corpus, using the TF-IDF technique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MOqeUNkg3w82",
    "outputId": "bfe3346d-713d-4114-b0e8-1a8539895072"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:300: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens [\"'d\", \"'ll\", \"'re\", \"'s\", \"'ve\", '``', 'al', 'au', 'bi', 'could', 'de', 'diesis', 'doe', 'dy', 'e', 'ha', 'might', 'mus', 'must', \"n't\", 'need', 'sha', 'un', 'wa', 'would'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "summarisation_processing_time_in_secs= 10.810009956359863\n",
      "summarised_text=\n",
      "[   'Envisioned as a three-year collaborative program, Smart Colab Program '\n",
      "    'will support around 100 institutions with AI infrastructure, course '\n",
      "    'content and curriculum, developer support, development tools and give '\n",
      "    'students access to cloud and AI services.',\n",
      "    'As part of the program, the Palo Alto giant which wants to expand its '\n",
      "    'reach and is planning to build a strong developer ecosystem in India with '\n",
      "    'the program will set up the core AI infrastructure and IoT Hub for the '\n",
      "    'selected campuses.']\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n"
     ]
    }
   ],
   "source": [
    "summarised_result = betterNLP.summarise(generic_text, method=\"tfidf\")\n",
    "\n",
    "print(\"~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\")\n",
    "print(\"summarisation_processing_time_in_secs=\",summarised_result['summarisation_processing_time_in_secs'])\n",
    "print(\"summarised_text=\")\n",
    "pp.pprint(summarised_result['summarised_text'])\n",
    "print(\"~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "important_words=\n",
      "[   ('ai', 0.5370250501018179),\n",
      "    ('program', 0.4130961923860138),\n",
      "    ('service', 0.2065480961930069),\n",
      "    ('skill', 0.2065480961930069),\n",
      "    ('build', 0.2065480961930069),\n",
      "    ('corp.', 0.16523847695440552),\n",
      "    ('smartsoft', 0.16523847695440552),\n",
      "    ('cloud', 0.12392885771580414),\n",
      "    ('require', 0.08261923847720276),\n",
      "    ('tomorrow', 0.08261923847720276),\n",
      "    ('job', 0.08261923847720276),\n",
      "    ('industry', 0.08261923847720276),\n",
      "    ('technology', 0.08261923847720276),\n",
      "    ('learning', 0.08261923847720276),\n",
      "    ('cognitive', 0.08261923847720276),\n",
      "    ('provide', 0.08261923847720276),\n",
      "    ('company', 0.08261923847720276),\n",
      "    ('india', 0.08261923847720276),\n",
      "    ('tool', 0.08261923847720276),\n",
      "    ('development', 0.08261923847720276),\n",
      "    ('developer', 0.08261923847720276),\n",
      "    ('course', 0.08261923847720276),\n",
      "    ('infrastructure', 0.08261923847720276),\n",
      "    ('institution', 0.08261923847720276),\n",
      "    ('support', 0.08261923847720276),\n",
      "    ('student', 0.08261923847720276),\n",
      "    ('ha', 0.08261923847720276),\n",
      "    ('colab', 0.08261923847720276),\n",
      "    ('smart', 0.08261923847720276),\n",
      "    ('announced', 0.08261923847720276),\n",
      "    ('workforce', 0.08261923847720276),\n",
      "    ('ai-ready', 0.08261923847720276),\n",
      "    ('attempt', 0.08261923847720276),\n",
      "    ('help', 0.04130961923860138),\n",
      "    ('asset', 0.04130961923860138),\n",
      "    ('bunch', 0.04130961923860138),\n",
      "    ('provided', 0.04130961923860138),\n",
      "    ('school', 0.04130961923860138),\n",
      "    ('developer-focused', 0.04130961923860138),\n",
      "    ('included', 0.04130961923860138),\n",
      "    ('well', 0.04130961923860138),\n",
      "    ('instructor', 0.04130961923860138),\n",
      "    ('expert', 0.04130961923860138),\n",
      "    ('lab', 0.04130961923860138),\n",
      "    ('hands-on', 0.04130961923860138),\n",
      "    ('featured', 0.04130961923860138),\n",
      "    ('online', 0.04130961923860138),\n",
      "    ('series', 0.04130961923860138),\n",
      "    ('science', 0.04130961923860138),\n",
      "    ('data', 0.04130961923860138),\n",
      "    ('hone', 0.04130961923860138),\n",
      "    ('wanted', 0.04130961923860138),\n",
      "    ('programmer', 0.04130961923860138),\n",
      "    ('ready', 0.04130961923860138),\n",
      "    ('developed', 0.04130961923860138),\n",
      "    ('wa', 0.04130961923860138),\n",
      "    ('public', 0.04130961923860138),\n",
      "    ('open', 0.04130961923860138),\n",
      "    ('track', 0.04130961923860138),\n",
      "    ('advanced', 0.04130961923860138),\n",
      "    ('year', 0.04130961923860138),\n",
      "    ('april', 0.04130961923860138),\n",
      "    ('earlier', 0.04130961923860138),\n",
      "    ('across', 0.04130961923860138),\n",
      "    ('application', 0.04130961923860138),\n",
      "    ('solution', 0.04130961923860138),\n",
      "    ('connected', 0.04130961923860138),\n",
      "    ('intelligent', 0.04130961923860138),\n",
      "    ('developing', 0.04130961923860138),\n",
      "    ('understanding', 0.04130961923860138),\n",
      "    ('in-depth', 0.04130961923860138),\n",
      "    ('aim', 0.04130961923860138),\n",
      "    (\"''\", 0.04130961923860138),\n",
      "    ('educate', 0.04130961923860138),\n",
      "    ('educator', 0.04130961923860138),\n",
      "    ('among', 0.04130961923860138),\n",
      "    ('capability', 0.04130961923860138),\n",
      "    ('set-up', 0.04130961923860138),\n",
      "    ('institutional', 0.04130961923860138),\n",
      "    ('ramp', 0.04130961923860138),\n",
      "    ('new', 0.04130961923860138),\n",
      "    ('integrate', 0.04130961923860138),\n",
      "    ('educational', 0.04130961923860138),\n",
      "    ('ever', 0.04130961923860138),\n",
      "    ('critical', 0.04130961923860138),\n",
      "    ('become', 0.04130961923860138),\n",
      "    ('’', 0.04130961923860138),\n",
      "    ('working', 0.04130961923860138),\n",
      "    ('training', 0.04130961923860138),\n",
      "    ('collaboration', 0.04130961923860138),\n",
      "    ('skillset', 0.04130961923860138),\n",
      "    ('different', 0.04130961923860138),\n",
      "    ('life', 0.04130961923860138),\n",
      "    ('transforming', 0.04130961923860138),\n",
      "    ('time', 0.04130961923860138),\n",
      "    ('defining', 0.04130961923860138),\n",
      "    ('``', 0.04130961923860138),\n",
      "    ('said', 0.04130961923860138),\n",
      "    ('manager', 0.04130961923860138),\n",
      "    ('country', 0.04130961923860138),\n",
      "    ('smith', 0.04130961923860138),\n",
      "    ('mark', 0.04130961923860138),\n",
      "    ('according', 0.04130961923860138),\n",
      "    ('machine', 0.04130961923860138),\n",
      "    ('bot', 0.04130961923860138),\n",
      "    ('campus', 0.04130961923860138),\n",
      "    ('selected', 0.04130961923860138),\n",
      "    ('hub', 0.04130961923860138),\n",
      "    ('iot', 0.04130961923860138),\n",
      "    ('core', 0.04130961923860138),\n",
      "    ('set', 0.04130961923860138),\n",
      "    ('ecosystem', 0.04130961923860138),\n",
      "    ('strong', 0.04130961923860138),\n",
      "    ('planning', 0.04130961923860138),\n",
      "    ('reach', 0.04130961923860138),\n",
      "    ('expand', 0.04130961923860138),\n",
      "    ('want', 0.04130961923860138),\n",
      "    ('giant', 0.04130961923860138),\n",
      "    ('alto', 0.04130961923860138),\n",
      "    ('palo', 0.04130961923860138),\n",
      "    ('part', 0.04130961923860138),\n",
      "    ('access', 0.04130961923860138),\n",
      "    ('give', 0.04130961923860138),\n",
      "    ('curriculum', 0.04130961923860138),\n",
      "    ('content', 0.04130961923860138),\n",
      "    ('100', 0.04130961923860138),\n",
      "    ('around', 0.04130961923860138),\n",
      "    ('collaborative', 0.04130961923860138),\n",
      "    ('three-year', 0.04130961923860138),\n",
      "    ('envisioned', 0.04130961923860138),\n",
      "    ('generation', 0.04130961923860138),\n",
      "    ('next', 0.04130961923860138),\n",
      "    ('empower', 0.04130961923860138),\n",
      "    ('launched', 0.04130961923860138)]\n"
     ]
    }
   ],
   "source": [
    "print(\"important_words=\")\n",
    "pp.pprint(summarised_result['important_words'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4jQLb4Sqfc0E"
   },
   "source": [
    "### 4. Summarising text in python using a variation of TF-IDF method\n",
    "\n",
    "\n",
    "Inspired by Shivangi Sareen from the posts:\n",
    "[Summarise Text with TFIDF in Python 1](https://towardsdatascience.com/tfidf-for-piece-of-text-in-python-43feccaa74f8) and [Summarise Text with TFIDF in Python 2](https://medium.com/@shivangisareen/summarise-text-with-tfidf-in-python-bc7ca10d3284)\n",
    "\n",
    "We have to break the text into sentences and tokens, ***we do not remove stop-words*** but do remove special characters. Tokenise words, calculate word TF and IDF frequencies to determine if a word is important on the corpus, using the TF-IDF technique. And then based on the average score method filter out only those sentences that meet the criteria.\n",
    "\n",
    "We could also use the (average score + 1.5 * std dev) or (average score + 3 * std dev), depending on the size of the target documents to summarise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BWAPOQyt3w87"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "summarisation_processing_time_in_secs= 0.027053356170654297\n",
      "('summarised_text=Earlier in April this year, the company announced SmartSoft '\n",
      " 'Corp. Advanced Program In AI as a learning track open to the public. That’s '\n",
      " 'why it has become more critical than ever for educational institutions to '\n",
      " 'integrate new cloud and AI technologies. According to Mark Smith, Country AI '\n",
      " 'Manager, SmartSoft Corp. India, said, \"With AI being the defining technology '\n",
      " 'of our time, it is transforming lives and industry and the jobs of tomorrow '\n",
      " 'will require a different skillset. In an attempt to build an AI-ready '\n",
      " 'workforce, SmartSoft Corp. announced Smart Colab Program which has been '\n",
      " 'launched to empower the next generation of students with AI-ready skills. '\n",
      " 'The program is an attempt to ramp up the institutional set-up and build '\n",
      " 'capabilities among the educators to educate the workforce of tomorrow.\"')\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n"
     ]
    }
   ],
   "source": [
    "summarised_result = betterNLP.summarise(generic_text, method=\"tfidf-ignore-stopwords\")\n",
    "\n",
    "print(\"~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\")\n",
    "print(\"summarisation_processing_time_in_secs=\",summarised_result['summarisation_processing_time_in_secs'])\n",
    "pp.pprint(\"summarised_text=\" + summarised_result['summarised_text'])\n",
    "print(\"~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scored_documents=\n",
      "[   {   'doc_id': 10,\n",
      "        'sent_score': 1.5890269151739729,\n",
      "        'sentence': 'Earlier in April this year, the company announced '\n",
      "                    'SmartSoft Corp. Advanced Program In AI as a learning '\n",
      "                    'track open to the public.'},\n",
      "    {   'doc_id': 7,\n",
      "        'sent_score': 1.285959013388815,\n",
      "        'sentence': 'That’s why it has become more critical than ever for '\n",
      "                    'educational institutions to integrate new cloud and AI '\n",
      "                    'technologies.'},\n",
      "    {   'doc_id': 5,\n",
      "        'sent_score': 1.0194666026064951,\n",
      "        'sentence': 'According to Mark Smith, Country AI Manager, SmartSoft '\n",
      "                    'Corp. India, said, \"With AI being the defining technology '\n",
      "                    'of our time, it is transforming lives and industry and '\n",
      "                    'the jobs of tomorrow will require a different skillset.'},\n",
      "    {   'doc_id': 1,\n",
      "        'sent_score': 0.9895530969743801,\n",
      "        'sentence': 'In an attempt to build an AI-ready workforce, SmartSoft '\n",
      "                    'Corp. announced Smart Colab Program which has been '\n",
      "                    'launched to empower the next generation of students with '\n",
      "                    'AI-ready skills.'},\n",
      "    {   'doc_id': 8,\n",
      "        'sent_score': 0.8573060089258766,\n",
      "        'sentence': 'The program is an attempt to ramp up the institutional '\n",
      "                    'set-up and build capabilities among the educators to '\n",
      "                    'educate the workforce of tomorrow.\"'},\n",
      "    {   'doc_id': 2,\n",
      "        'sent_score': 0.5921245466315593,\n",
      "        'sentence': 'Envisioned as a three-year collaborative program, Smart '\n",
      "                    'Colab Program will support around 100 institutions with '\n",
      "                    'AI infrastructure, course content and curriculum, '\n",
      "                    'developer support, development tools and give students '\n",
      "                    'access to cloud and AI services.'},\n",
      "    {   'doc_id': 3,\n",
      "        'sent_score': 0.3900792787747875,\n",
      "        'sentence': 'As part of the program, the Palo Alto giant which wants '\n",
      "                    'to expand its reach and is planning to build a strong '\n",
      "                    'developer ecosystem in India with the program will set up '\n",
      "                    'the core AI infrastructure and IoT Hub for the selected '\n",
      "                    'campuses.'},\n",
      "    {   'doc_id': 9,\n",
      "        'sent_score': 0.3900792787747875,\n",
      "        'sentence': 'The program aims to build up the cognitive skills and '\n",
      "                    'in-depth understanding of developing intelligent cloud '\n",
      "                    'connected solutions for applications across industry.'},\n",
      "    {   'doc_id': 11,\n",
      "        'sent_score': 0.3900792787747875,\n",
      "        'sentence': 'The program was developed to provide job ready skills to '\n",
      "                    'programmers who wanted to hone their skills in AI and '\n",
      "                    'data science with a series of online courses which '\n",
      "                    'featured hands-on labs and expert instructors as well.'},\n",
      "    {   'doc_id': 12,\n",
      "        'sent_score': 0.3900792787747875,\n",
      "        'sentence': 'This program also included developer-focused AI school '\n",
      "                    'that provided a bunch of assets to help build AI skills.'},\n",
      "    {   'doc_id': 4,\n",
      "        'sent_score': 0.28905664484640153,\n",
      "        'sentence': 'The company will provide AI development tools and AI '\n",
      "                    'services such as SmartSoft Corp. Cognitive Services, Bot '\n",
      "                    'Services and Machine Learning Services.'},\n",
      "    {   'doc_id': 6,\n",
      "        'sent_score': 0.0870113769896297,\n",
      "        'sentence': 'This will require more collaborations and training and '\n",
      "                    'working with AI.'}]\n"
     ]
    }
   ],
   "source": [
    "print(\"scored_documents=\")\n",
    "pp.pprint(summarised_result['scored_documents'])"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "better_nlp_summarisers.ipynb",
   "version": "0.3.2",
   "provenance": [],
   "collapsed_sections": [],
   "include_colab_link": true
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
