{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# import2vec - Learning Embeddings for SW Libraries\n",
    "\n",
    "(see paper https://arxiv.org/abs/1904.03990)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please first select the **language** for which you want to train library embeddings. More configuration options are available in the config section below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "language = 'javascript'   # Supported languages: python, javascript, java\n",
    "\n",
    "max_projects = 10000        # Maximum number of GitHub projects to download. Set to 0 to download all.\n",
    "min_stars = 10            # Minimum number of GitHub stars a project should have to be considered.\n",
    "max_size = 0              # Maximum project size (in kb). Set to 0 for unlimitted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Library Vectors\n",
    "\n",
    "Trains vectors for Python libraries (top-level modules). Python imports are first mapped to the top-level package they are defined in. Then, the model is trained by creating target/context pairs for any 2 Python packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Setup Environment & Download Dataset from GitHub\n",
    "\n",
    "First let's check your environment, install required libraries (if needed) and download a dataset for training.\n",
    "\n",
    "⚠️ The dataset required for training Library Vectors will be downloaded through the GitHub API. You therefore first need to request an <font color='green'>API key</font> at https://github.com/settings/tokens and write it down in a file called <font color='green'>apikey.txt</font> in the <font color='green'>scripts</font> sub-directory. See https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line for more information on how to create an API key (personal access token). In step 7, you need to request the following scope: <font color='green'>repo > public_repo</font>.\n",
    "\n",
    "⚠️ Make sure your jupyter environment is setup correctly. If you run into issues, please execute the following commands and restart your jupyter or jupyter lab environment:\n",
    "\n",
    "```bash\n",
    "     jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
    "```\n",
    "\n",
    "To have a realistic dataset, you will have to configure the below command line arguments: \n",
    "\n",
    "```bash\n",
    "   run.sh <language> [<maxprojects>] [<minstars>] [<maxsize>] [<usecache>]\n",
    "       \n",
    "   where\n",
    "     - <language>      Programming language. Supported languages: python, java, javascript, csharp, php, ruby\n",
    "     - <maxprojects>   Maximum GitHub projects (default = 0: all)\n",
    "     - <minstars>      Min GitHub stars (default = 2)\n",
    "     - <maxsize>       Max project size (in kb) (default = 0)\n",
    "     - <usecache>      Cache intermediate files/scripts (default = 0)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "supported_languages = ['python', 'java', 'javascript']\n",
    "if not language in supported_languages:\n",
    "    raise ValueError('Unsupported language: ' + language)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! cd ../scripts; ./run.sh $language $max_projects $min_stars $max_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install keras json_lines sklearn gensim scipy matplotlib tensorflow bcolz bitarray ipywidgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-13T09:35:20.289063Z",
     "start_time": "2019-01-13T09:35:19.028491Z"
    }
   },
   "outputs": [],
   "source": [
    "import gc\n",
    "import gzip\n",
    "import collections\n",
    "import math\n",
    "import os\n",
    "import sys\n",
    "import zipfile\n",
    "from pathlib import Path\n",
    "import json\n",
    "import string\n",
    "from six import string_types\n",
    "\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, Reshape, merge, dot, add, BatchNormalization\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from sklearn.manifold import TSNE as tsne\n",
    "from gensim import matutils\n",
    "from scipy.spatial.distance import cosine\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "from tqdm import tqdm_notebook as tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Configuration\n",
    "\n",
    "You can change the below parameters to tune the vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-13T09:35:21.799689Z",
     "start_time": "2019-01-13T09:35:21.794905Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'../datasets/javascript/'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rootDir = '../datasets/' + language + '/'\n",
    "config = {}\n",
    "\n",
    "config['validation_ratio'] = 0.0         # number between 0 and 1 corresponding to the percentage of projects to hold out for validation.\n",
    "config['true_negative_sampling'] = True  # if True, check that negative samples never co-occur in the entire dataset, by using a co-occurrence matrix.\n",
    "config['negative_sampling_ratio'] = 1    # number of negative samples to generate for each positive sample.\n",
    "\n",
    "# tuning\n",
    "config['reduce_dict_size'] = True        # if True, libraries will first be filtered before calculating vectors\n",
    "\n",
    "config['min_import_freq'] = 2            # minimum number of times a library must be imported for it to be kept in the vocabulary\n",
    "config['max_import_freq'] = 1000000      # maximum number of times a library can be imported for it to be kept in the vocabulary\n",
    "config['min_imports'] = 2                # minimum number of imports that must be present in any source file for the file to be kept in the dataset.\n",
    "\n",
    "# Model parameters\n",
    "config['window_size'] = 1000             # a large window size gives less importance to the sequence in which libraries are imported in source files.\n",
    "config['vector_dim'] = 100               # the dimensionality of the trained library vectors.\n",
    "config['epochs'] = 100                   # number of epochs to train.\n",
    "\n",
    "rootDir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir -p $rootDir/models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-13T09:35:28.905794Z",
     "start_time": "2019-01-13T09:35:24.386454Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7891 JAVASCRIPT files read"
     ]
    }
   ],
   "source": [
    "directory = rootDir + '/processed/'\n",
    "\n",
    "datasets = {}\n",
    "for filename in filter(lambda x: x.endswith('.json.gz'),os.listdir(directory)):\n",
    "    with gzip.open(directory + filename, 'rt') as f:\n",
    "        #print(\"Loading \" + filename + \"...\")\n",
    "        new_dataset = json.loads(f.read())\n",
    "        datasets = {**datasets, **new_dataset}\n",
    "        sys.stdout.write('\\r' + str(len(datasets)) + ' ' + language.upper() + ' files read')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:15.796987Z",
     "start_time": "2019-01-10T08:28:15.783025Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7891,\n",
       " ['../datasets/javascript/dataset01/0123cf/auto-layout',\n",
       "  '../datasets/javascript/dataset01/0123cf/together-draw',\n",
       "  '../datasets/javascript/dataset01/06wj/pokemon',\n",
       "  '../datasets/javascript/dataset01/07Gond/npm-global-rm-interactive',\n",
       "  '../datasets/javascript/dataset01/0age/metamorphic'])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "projects = [k for k,v in datasets.items()]\n",
    "len(projects), projects[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:20.142325Z",
     "start_time": "2019-01-10T08:28:15.798009Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# utility functions for the supported languages\n",
    "\n",
    "def to_lib(imp):      # converts imp to library part (i.e. ignore everything after .)\n",
    "    return imp.split('.')[0].split(':')[0]\n",
    "\n",
    "def to_package(imp):  # converts imp to package\n",
    "    return '.'.join(imp.split('.')[:-1])\n",
    "\n",
    "# for javascript (NPM)\n",
    "accepted_first_chars = string.ascii_letters + string.digits + '@'\n",
    "\n",
    "is_valid = lambda x: True\n",
    "\n",
    "if language == 'python':\n",
    "    mapping = to_lib\n",
    "elif language == 'java':\n",
    "    mapping = to_package\n",
    "elif language == 'javascript':\n",
    "    mapping = lambda x: x\n",
    "    is_valid = lambda x: x[0] in accepted_first_chars\n",
    "else:\n",
    "    print(\"Unsupported language:\", language)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:20.142325Z",
     "start_time": "2019-01-10T08:28:15.798009Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f013175964e04902b03bc0e46bbf02f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=7891), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "data = []   # list of (project, [project imports])\n",
    "\n",
    "for project, files in tqdm(datasets.items()):\n",
    "    data.append([project, list(set([mapping(imp) for file, imps in files.items() for imp in imps if is_valid(imp)]))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:20.163338Z",
     "start_time": "2019-01-10T08:28:20.143367Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7592"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove empty projects\n",
    "data = list(filter(lambda row: len(row[1]) > 0, data))\n",
    "len(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:20.467329Z",
     "start_time": "2019-01-10T08:28:20.164495Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "94edf4c107af43edb1de312d67e1522a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=7592), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['../datasets/javascript/dataset01/0123cf/auto-layout',\n",
       " ['html-webpack-plugin', 'path', 'clean-webpack-plugin']]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# filter imps (remove duplicates and empty)\n",
    "def remove_empty(imps):\n",
    "    if '' in imps:\n",
    "        imps.remove('')\n",
    "    return imps\n",
    "\n",
    "data = [[row[0], remove_empty(list(set(row[1])))] for row in tqdm(data)]\n",
    "data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Create Training and Validation Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:20.485180Z",
     "start_time": "2019-01-10T08:28:20.468835Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7592, 7592, 0)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.shuffle(data)\n",
    "split_point = math.ceil(len(data) * (1-config['validation_ratio']))\n",
    "data_trn = data[:split_point]\n",
    "data_val = data[split_point:]\n",
    "len(data), len(data_trn), len(data_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:20.498443Z",
     "start_time": "2019-01-10T08:28:20.486186Z"
    }
   },
   "outputs": [],
   "source": [
    "data = None # can't use data anymore, must use data_trn or data_val from this point onwards!\n",
    "del data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1 Save the Training and Validation Set to disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:20.628066Z",
     "start_time": "2019-01-10T08:28:20.500917Z"
    }
   },
   "outputs": [],
   "source": [
    "import bcolz\n",
    "  \n",
    "def save_array(fname, arr):\n",
    "    c=bcolz.carray(arr, rootdir=fname, mode='w')\n",
    "    c.flush()\n",
    "\n",
    "def load_array(fname):\n",
    "    return bcolz.open(fname)[:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:22.009171Z",
     "start_time": "2019-01-10T08:28:20.631258Z"
    }
   },
   "outputs": [],
   "source": [
    "with open(rootDir + 'models/data_trn_dim{}.json'.format(config['vector_dim']), 'w', encoding='utf-8') as out:\n",
    "    json.dump(data_trn, out)\n",
    "if len(data_val) > 0:\n",
    "    with open(rootDir + 'models/data_val_dim{}.json'.format(config['vector_dim']), 'w', encoding='utf-8') as out:\n",
    "        json.dump(data_val, out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2 Load Validation and Training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:22.842327Z",
     "start_time": "2019-01-10T08:28:22.010394Z"
    }
   },
   "outputs": [],
   "source": [
    "with open(rootDir + 'models/data_trn_dim{}.json'.format(config['vector_dim']), 'r', encoding='utf-8') as f:\n",
    "    data_trn = json.load(f)\n",
    "if config['validation_ratio'] > 0.0:\n",
    "    with open(rootDir + 'models/data_val_dim{}.json'.format(config['vector_dim']), 'r', encoding='utf-8') as f:\n",
    "        data_val = json.load(f)\n",
    "else:\n",
    "    data_val = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.3 Build dictionary of imports\n",
    "\n",
    "The dictionary maps each imported library to an index. The index will later be used as the position in the imports vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's create list of (class/project, import) pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:24.021571Z",
     "start_time": "2019-01-10T08:28:22.843474Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "132553"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = [[row[0], imp] for row in data_trn for imp in row[1]]\n",
    "len(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We transform the dataset into a dictionary of library name to a dictionary with library frequency and index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:24.990521Z",
     "start_time": "2019-01-10T08:28:24.022664Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "104533e9e69a49b4b484175843b708c2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=132553), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "idx = 0;\n",
    "lib_dict = {}\n",
    "for row in tqdm(dataset):\n",
    "    lib = row[1]\n",
    "    if lib in lib_dict:\n",
    "        lib_dict[lib]['count'] += 1\n",
    "    else:\n",
    "        lib_dict[lib] = {'count': 1, 'index': idx}\n",
    "        idx += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a total vocabulary size (number of distinct imports) of:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:25.111796Z",
     "start_time": "2019-01-10T08:28:25.101345Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "35447"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_size = len(lib_dict)\n",
    "vocab_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "on a total number of imports of:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-10-12T13:08:03.582867Z",
     "start_time": "2018-10-12T13:08:03.570867Z"
    }
   },
   "source": [
    "del dataset\n",
    "del data_val\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.4 Reduce the size of the dictionary\n",
    "\n",
    "In order to make the dataset managable for visulization, we can experiment with a few heuristics:\n",
    "- remove libraries that aren't frequently imported: configured with `min_import_freq`\n",
    "- remove libraries that are too frequently imported: configured with `max_import_freq`\n",
    "- remove classes/projects that have too few imports (from the set of reduced imports): `min_imports`\n",
    "\n",
    "The reasoning is the following:\n",
    "- if a library is only imported a few times, they can not contribute to clusters of significant size.\n",
    "- too frequently imported libraries are very generic and therefore not distinctive enough for clustering. As an example, libraries that do logging or the standard java libraries everyone imports like `java.lang` and `java.util`.\n",
    "- if there are not enough imports left, there is not much association with other projects that can be done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:25.159775Z",
     "start_time": "2019-01-10T08:28:25.112690Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9303\n"
     ]
    }
   ],
   "source": [
    "if config['reduce_dict_size']:\n",
    "    lib_dict = {k:v for k,v in lib_dict.items() if v['count'] >= config['min_import_freq'] and v['count'] <= config['max_import_freq']}\n",
    "    vocab_size = len(lib_dict)\n",
    "    print(vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to re-index `lib_dict`, so that the indices fall within `0..vocab_size`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:25.203220Z",
     "start_time": "2019-01-10T08:28:25.160842Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['util', 'child_process', 'dotenv', 'request-promise', 'path'],\n",
       " [0, 1, 2, 3, 4])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lib_dict = {k: {\"index\": idx, \"freq\": lib_dict[k][\"count\"]} for idx, k in enumerate(lib_dict)}\n",
    "reversed_lib_dict = dict(zip([v[\"index\"] for v in lib_dict.values()], lib_dict.keys()))\n",
    "list(lib_dict)[:5], list(reversed_lib_dict)[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.5 Save the Library dictionary to disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:25.466497Z",
     "start_time": "2019-01-10T08:28:25.204440Z"
    }
   },
   "outputs": [],
   "source": [
    "with open(rootDir + 'models/lib_dict_dim{}.json'.format(config['vector_dim']), 'w', encoding='utf-8') as out:\n",
    "    json.dump(lib_dict, out)\n",
    "\n",
    "with open(rootDir + 'models/reversed_lib_dict_dim{}.json'.format(config['vector_dim']), 'w', encoding='utf-8') as out:\n",
    "    json.dump(reversed_lib_dict, out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:25.471574Z",
     "start_time": "2019-01-10T08:28:25.467812Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9303, ['util', 'child_process', 'dotenv', 'request-promise', 'path'])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lib_names = list(lib_dict)\n",
    "len(lib_names), lib_names[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:26.655579Z",
     "start_time": "2019-01-10T08:28:25.472489Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6527264f66084342bc9fbbdb6b131947",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=7592), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "data_dict_reduced = {}\n",
    "for row in tqdm(data_trn):\n",
    "    if row[0] in data_dict_reduced:\n",
    "        print(\"Ignoring duplicate project:\", row[0])\n",
    "    else:\n",
    "        filtered_libs = [lib for lib in row[1] if lib in lib_dict]\n",
    "        if len(filtered_libs) > config['min_imports']:\n",
    "            lib_indices = [lib_dict[lib][\"index\"] for lib in filtered_libs]\n",
    "            data_dict_reduced[row[0]] = { \"libs\": filtered_libs, \"lib_ids\": lib_indices }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:26.659076Z",
     "start_time": "2019-01-10T08:28:26.656711Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6298"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_dict_reduced)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:26.675590Z",
     "start_time": "2019-01-10T08:28:26.659993Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6298"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_list = [v['lib_ids'] for _, v in data_dict_reduced.items()]\n",
    "len(data_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:26.717631Z",
     "start_time": "2019-01-10T08:28:26.687968Z"
    }
   },
   "outputs": [],
   "source": [
    "del data_dict_reduced"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.6 Build co-occurence matrix for True negative sampling\n",
    "\n",
    "Calculate the co-occurence frequency of libraries\n",
    "\n",
    "Space-efficient storage of co-occurrence matrix:\n",
    "\n",
    "Store in flattened version of the upper triangle of a matrix of size nxn.\n",
    "Example: co(0,4) + co(1,2) + co(3,2) + co(3,4) gives\n",
    "\n",
    "    .0001\n",
    "    ..100\n",
    "    ...10\n",
    "    ....1\n",
    "    .....\n",
    "\n",
    "The formula to find the index in the flattened n-dimensional bit-array for storing the co-occurrence of (i,j) is given by $(i.n) + j - \\bigg(\\frac{(i+1).(i+2)}{2}\\bigg)$, where $i < j$  \n",
    "The size of the bitarray is given by $ \\frac{(n-1).n}{2} $\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:26.730605Z",
     "start_time": "2019-01-10T08:28:26.718821Z"
    }
   },
   "outputs": [],
   "source": [
    "def co_idx(i, j, n):\n",
    "    if i == j:\n",
    "        print(\"indices should be different\")\n",
    "        return -1\n",
    "    (s, l) = (i, j) if i < j else (j, i)\n",
    "    return int(s * n + l - (s+1)*(s+2)/2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:26.743450Z",
     "start_time": "2019-01-10T08:28:26.731622Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.005037087597884238 GB\n"
     ]
    }
   ],
   "source": [
    "co_size = (vocab_size-1)*vocab_size/2\n",
    "print(co_size / 1024/1024/1024/8, 'GB')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:40.640811Z",
     "start_time": "2019-01-10T08:28:26.744452Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "36635ba4d5ce4e0da6648046882b11f5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=6298), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "1147263\n"
     ]
    }
   ],
   "source": [
    "from bitarray import bitarray\n",
    "\n",
    "if config['true_negative_sampling']:\n",
    "    co_occurrence = bitarray(int(co_size))\n",
    "    co_occurrence.setall(False)\n",
    "    for row in tqdm(data_list):\n",
    "        for i in range(0, len(row)-1):\n",
    "            for j in range(i+1, len(row)):\n",
    "                co_occurrence[co_idx(row[i], row[j], vocab_size)] = True\n",
    "    print(co_occurrence.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.7 Build file-level sequences of library imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:47.900111Z",
     "start_time": "2019-01-10T08:28:40.642091Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "94bae1f590874c13af89368f139f1546",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=7891), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "file_data = []\n",
    "\n",
    "for project, files in tqdm(datasets.items()):\n",
    "    for file, imps in files.items():\n",
    "        file_imps = list(set([mapping(imp) for imp in imps if is_valid(imp)]))\n",
    "        if len(file_imps) > 0:\n",
    "            file_data.append([project + ':' + file, file_imps])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:47.903871Z",
     "start_time": "2019-01-10T08:28:47.901247Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['../datasets/javascript/dataset01/0123cf/auto-layout:webpack.config.js',\n",
       "  ['html-webpack-plugin', 'path', 'clean-webpack-plugin']],\n",
       " ['../datasets/javascript/dataset01/0123cf/together-draw:server/index.js',\n",
       "  ['http', 'socket.io', 'express']],\n",
       " ['../datasets/javascript/dataset01/0123cf/together-draw:src/index.js',\n",
       "  ['fabric']],\n",
       " ['../datasets/javascript/dataset01/0123cf/together-draw:webpack.config.dev.js',\n",
       "  ['html-webpack-plugin', 'path', 'webpack']],\n",
       " ['../datasets/javascript/dataset01/0123cf/together-draw:webpack.config.prod.js',\n",
       "  ['html-webpack-plugin', 'path', 'webpack']]]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_data[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:53.130862Z",
     "start_time": "2019-01-10T08:28:47.904740Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df832e5806b04de0b718695f0a528a66",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=189370), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "file_data_dict = {}\n",
    "for row in tqdm(file_data):\n",
    "    if row[0] in file_data_dict:\n",
    "        print(\"Ignoring duplicate file:\", row[0])\n",
    "    else:\n",
    "        filtered_libs = [lib for lib in row[1] if lib in lib_dict]\n",
    "        if len(filtered_libs) > 0:\n",
    "            lib_indices = [lib_dict[lib][\"index\"] for lib in filtered_libs]\n",
    "            file_data_dict[row[0]] = { \"libs\": filtered_libs, \"lib_ids\": lib_indices }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:53.299630Z",
     "start_time": "2019-01-10T08:28:53.132027Z"
    }
   },
   "outputs": [],
   "source": [
    "del file_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:53.319106Z",
     "start_time": "2019-01-10T08:28:53.300837Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "174607"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(file_data_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:53.415255Z",
     "start_time": "2019-01-10T08:28:53.320004Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "174607"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_data_list = [v['lib_ids'] for _, v in file_data_dict.items()]\n",
    "len(file_data_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:53.668274Z",
     "start_time": "2019-01-10T08:28:53.419595Z"
    }
   },
   "outputs": [],
   "source": [
    "del file_data_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.8 Generate Context-Target Library Pairs for each File-level Sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:53.702348Z",
     "start_time": "2019-01-10T08:28:53.669462Z"
    },
    "code_folding": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total count: 106409\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import math\n",
    "\n",
    "def make_sampling_table(lib_dict, sampling_factor=1e-4):\n",
    "    total_count = 0\n",
    "    for _,v in lib_dict.items():\n",
    "        total_count += v[\"freq\"]\n",
    "    print(\"total count:\", total_count)\n",
    "    return [min(1, math.sqrt(sampling_factor/(v[\"freq\"]/total_count))) for _,v in lib_dict.items()]\n",
    "    \n",
    "sampling_table = make_sampling_table(lib_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:53.787905Z",
     "start_time": "2019-01-10T08:28:53.703557Z"
    },
    "code_folding": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9303, array([['path', '0.05346886884457664'],\n",
       "        ['fs', '0.06349938013082382'],\n",
       "        ['react', '0.06716224589952868'],\n",
       "        ['react-dom', '0.08914539063879517'],\n",
       "        ['webpack', '0.10183932228443406'],\n",
       "        ['express', '0.10274457850757096'],\n",
       "        ['child_process', '0.10279558103168882'],\n",
       "        ['prop-types', '0.11755573422573078'],\n",
       "        ['util', '0.1179392776076502'],\n",
       "        ['html-webpack-plugin', '0.11983400574909088']], dtype='<U66'))"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampling_probs = np.array(list(zip(list(lib_dict),sampling_table)))\n",
    "sorted_sampling_probs = sampling_probs[np.argsort(sampling_probs[:,1])]\n",
    "len(sorted_sampling_probs), sorted_sampling_probs[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:53.793363Z",
     "start_time": "2019-01-10T08:28:53.789008Z"
    },
    "code_folding": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[7, 9, 3, 5, 8]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def samples(arr, num, sampling_table):\n",
    "    \"\"\"\n",
    "        Uses sub-sampling to return num samples (library indices) from the given array of \n",
    "        library indices using the given library frequency table.\n",
    "        # Arguments\n",
    "        arr: the array of possible indices to sample from\n",
    "        num: the number of samples to return\n",
    "        sampling_table: a list of size vocab_size with sampling rates for each library, being the \n",
    "            probability with which that library should be chosen.\n",
    "    \"\"\"\n",
    "    res = []\n",
    "    i = 0\n",
    "    while len(res) < num:\n",
    "        if sampling_table[arr[i]] >= random.random() and not arr[i] in res:\n",
    "            res.append(arr[i])            \n",
    "        i += 1\n",
    "        if i >= len(arr):\n",
    "            i = 0\n",
    "    return res\n",
    "\n",
    "samples(range(10), 5, sampling_table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:28:53.806052Z",
     "start_time": "2019-01-10T08:28:53.794350Z"
    },
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "def randomgrams(sequences, vocabulary_size, num_positive_samples=8, negative_sampling_ratio=1, true_negative_sampling=False,\n",
    "                sampling_table=sampling_table, shuffle=True):\n",
    "    \"\"\"\n",
    "        Generates randomgram library pairs. This is similar to skipgrams, but we don't take\n",
    "        a sliding window, since we are dealing with unordered data. We simply take num_positive_samples\n",
    "        libraries out of a bag of given libraries that are imported by a certain project.\n",
    "        \n",
    "        Takes a list of sequences (list of indexes of libraries),\n",
    "        returns couples of [lib_index, other_lib_index] and labels (1s or 0s),\n",
    "        where label = 1 if 'other_lib' belongs to the context of 'lib' in the sequence,\n",
    "        and label=0 if 'other_lib' is randomly sampled\n",
    "        # Arguments\n",
    "        sequence: a library sequence (bag), encoded as a list\n",
    "            of library indices (integers). If using a `sampling_table`,\n",
    "            library indices are expected to match the rank\n",
    "            of the libraries in a reference dataset (e.g. 10 would encode\n",
    "            the 10-th most frequently occurring token).\n",
    "            Note that index 0 is expected to be a non-library and will be skipped.\n",
    "        vocabulary_size: int. maximum possible word index + 1\n",
    "        num_positive_samples: int. number of positive samples to generate pairs for.\n",
    "        negative_sample_ratio: float. ratio of negative samples compared to the amount of positive samples.\n",
    "        true_negative_sampling: if True, verify that negative sample is truely negative by crosschecking in co-occurrence matrix.\n",
    "        shuffle: whether to shuffle the couples before returning them.\n",
    "        # Returns\n",
    "        couples, labels: where `couples` are int pairs and\n",
    "            `labels` are either 0 or 1.\n",
    "    \"\"\"\n",
    "    target_words = []\n",
    "    context_words = []\n",
    "    labels = []\n",
    "    conflicts = 0\n",
    "    for sequence in tqdm(sequences):\n",
    "        for i, li in enumerate(sequence):\n",
    "            if not li:\n",
    "                continue\n",
    "            num = min(len(sequence)-1,num_positive_samples)\n",
    "            if sampling_table:\n",
    "                for lj in samples(sequence[:i] + sequence[i+1:], num, sampling_table):\n",
    "                    target_words.append(li)\n",
    "                    context_words.append(lj)\n",
    "                    labels.append(1)\n",
    "            else:\n",
    "                for j in range(num):\n",
    "                    if j != i:\n",
    "                        lj = sequence[j]\n",
    "                        if not lj:\n",
    "                            continue\n",
    "                        target_words.append(li)\n",
    "                        context_words.append(lj)\n",
    "                        labels.append(1)\n",
    "            if negative_sampling_ratio != 0:\n",
    "                num_neg = math.floor(num * negative_sampling_ratio)\n",
    "                neg_samples = random.sample(range(0, vocabulary_size-1), num_neg)\n",
    "                for neg_sample in neg_samples:\n",
    "                    if true_negative_sampling:\n",
    "                        while li != neg_sample and co_occurrence[co_idx(li, neg_sample, vocab_size)]:\n",
    "                            conflicts += 1\n",
    "                            neg_sample = random.randint(0, vocabulary_size-1)\n",
    "                    target_words.append(li)\n",
    "                    context_words.append(neg_sample)\n",
    "                    labels.append(0)\n",
    "                \n",
    "    if shuffle:\n",
    "        print('shuffling...')\n",
    "        seed = random.randint(0, 10e6)\n",
    "        random.seed(seed)\n",
    "        random.shuffle(target_words)\n",
    "        random.seed(seed)\n",
    "        random.shuffle(context_words)\n",
    "        random.seed(seed)\n",
    "        random.shuffle(labels)\n",
    "\n",
    "    if true_negative_sampling:\n",
    "        print(\"conflicts =\", conflicts)\n",
    "    return target_words, context_words, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:34:14.139617Z",
     "start_time": "2019-01-10T08:28:53.809261Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f4be3be5a914c9ba2ebf7025b499d55",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=174607), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "shuffling...\n",
      "conflicts = 1457327\n",
      "#couples = 4129262\n"
     ]
    }
   ],
   "source": [
    "lib_target, lib_context, labels = randomgrams(file_data_list, vocab_size, num_positive_samples=config['window_size'], negative_sampling_ratio=config['negative_sampling_ratio'], true_negative_sampling=config['true_negative_sampling'])\n",
    "print(\"#couples =\", len(lib_target))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:34:14.144761Z",
     "start_time": "2019-01-10T08:34:14.141001Z"
    }
   },
   "outputs": [],
   "source": [
    "del co_occurrence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:34:21.375067Z",
     "start_time": "2019-01-10T08:34:14.145763Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5275  607 6699  637 8743   73  101 8777  254 4339   98  183 3141  676\n",
      "  101  112   27 5225  101 7764]\n",
      "[4885 6969  101  103 8714 1592  423 5072 5261 7683 1511 2848 3218 2354\n",
      " 4725  117 3171 5279 1374 4574]\n",
      "[0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "lib_target = np.array(lib_target, dtype=\"int32\")\n",
    "lib_context = np.array(lib_context, dtype=\"int32\")\n",
    "\n",
    "print(lib_target[:20])\n",
    "print(lib_context[:20])\n",
    "print(labels[:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:34:23.072833Z",
     "start_time": "2019-01-10T08:34:21.376276Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2064631, 4129262)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(labels), len(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:34:24.788780Z",
     "start_time": "2019-01-10T08:34:23.073858Z"
    }
   },
   "outputs": [],
   "source": [
    "labels = np.array(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Train the Library Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.1 Construct the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:34:24.801826Z",
     "start_time": "2019-01-10T08:34:24.789920Z"
    }
   },
   "outputs": [],
   "source": [
    "use_bias = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T08:42:53.719159Z",
     "start_time": "2019-01-10T08:42:53.656002Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/theetenb/anaconda/envs/test/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    }
   ],
   "source": [
    "# create some input variables\n",
    "input_target = Input((1,))\n",
    "input_context = Input((1,))\n",
    "\n",
    "embedding = Embedding(vocab_size, config['vector_dim'], input_length=1, name='embedding')\n",
    "target = embedding(input_target)\n",
    "target = Reshape((config['vector_dim'], 1))(target)\n",
    "context = embedding(input_context)\n",
    "context = Reshape((config['vector_dim'], 1))(context)\n",
    "\n",
    "# now perform the dot product operation to get a similarity measure\n",
    "dot_product = dot([target, context], 1, normalize=True)\n",
    "dot_product = Reshape((1,))(dot_product)\n",
    "\n",
    "# add the sigmoid output layer\n",
    "output = Dense(1, activation='sigmoid', use_bias=use_bias)(dot_product)\n",
    "\n",
    "# create the primary training model\n",
    "model = Model(inputs=[input_target, input_context], outputs=output)\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='rmsprop')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.2 Train the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2019-01-10T14:01:28.822Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch size: 32768\n",
      "WARNING:tensorflow:From /home/theetenb/anaconda/envs/test/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Epoch 1/100\n",
      " - 9s - loss: 0.5272\n",
      "Epoch 2/100\n",
      " - 8s - loss: 0.4325\n",
      "Epoch 3/100\n",
      " - 8s - loss: 0.4080\n",
      "Epoch 4/100\n",
      " - 9s - loss: 0.3883\n",
      "Epoch 5/100\n",
      " - 8s - loss: 0.3706\n",
      "Epoch 6/100\n",
      " - 8s - loss: 0.3542\n",
      "Epoch 7/100\n",
      " - 8s - loss: 0.3387\n",
      "Epoch 8/100\n",
      " - 8s - loss: 0.3242\n",
      "Epoch 9/100\n",
      " - 8s - loss: 0.3105\n",
      "Epoch 10/100\n",
      " - 8s - loss: 0.2975\n",
      "Epoch 11/100\n",
      " - 8s - loss: 0.2853\n",
      "Epoch 12/100\n",
      " - 8s - loss: 0.2736\n",
      "Epoch 13/100\n",
      " - 8s - loss: 0.2626\n",
      "Epoch 14/100\n",
      " - 8s - loss: 0.2522\n",
      "Epoch 15/100\n",
      " - 8s - loss: 0.2423\n",
      "Epoch 16/100\n",
      " - 9s - loss: 0.2329\n",
      "Epoch 17/100\n",
      " - 8s - loss: 0.2241\n",
      "Epoch 18/100\n",
      " - 8s - loss: 0.2156\n",
      "Epoch 19/100\n",
      " - 8s - loss: 0.2076\n",
      "Epoch 20/100\n",
      " - 8s - loss: 0.2001\n",
      "Epoch 21/100\n",
      " - 9s - loss: 0.1929\n",
      "Epoch 22/100\n",
      " - 8s - loss: 0.1861\n",
      "Epoch 23/100\n",
      " - 8s - loss: 0.1796\n",
      "Epoch 24/100\n",
      " - 8s - loss: 0.1734\n",
      "Epoch 25/100\n",
      " - 8s - loss: 0.1676\n",
      "Epoch 26/100\n",
      " - 8s - loss: 0.1620\n",
      "Epoch 27/100\n",
      " - 8s - loss: 0.1567\n",
      "Epoch 28/100\n",
      " - 8s - loss: 0.1517\n",
      "Epoch 29/100\n",
      " - 8s - loss: 0.1469\n",
      "Epoch 30/100\n",
      " - 8s - loss: 0.1424\n",
      "Epoch 31/100\n",
      " - 8s - loss: 0.1381\n",
      "Epoch 32/100\n",
      " - 8s - loss: 0.1339\n",
      "Epoch 33/100\n",
      " - 8s - loss: 0.1300\n",
      "Epoch 34/100\n",
      " - 8s - loss: 0.1263\n",
      "Epoch 35/100\n",
      " - 8s - loss: 0.1227\n",
      "Epoch 36/100\n",
      " - 8s - loss: 0.1193\n",
      "Epoch 37/100\n",
      " - 8s - loss: 0.1160\n",
      "Epoch 38/100\n",
      " - 8s - loss: 0.1128\n",
      "Epoch 39/100\n",
      " - 8s - loss: 0.1098\n",
      "Epoch 40/100\n",
      " - 8s - loss: 0.1070\n",
      "Epoch 41/100\n",
      " - 8s - loss: 0.1042\n",
      "Epoch 42/100\n",
      " - 8s - loss: 0.1016\n",
      "Epoch 43/100\n",
      " - 9s - loss: 0.0990\n",
      "Epoch 44/100\n",
      " - 8s - loss: 0.0966\n",
      "Epoch 45/100\n",
      " - 8s - loss: 0.0943\n",
      "Epoch 46/100\n",
      " - 8s - loss: 0.0920\n",
      "Epoch 47/100\n",
      " - 8s - loss: 0.0898\n",
      "Epoch 48/100\n",
      " - 8s - loss: 0.0877\n",
      "Epoch 49/100\n",
      " - 8s - loss: 0.0857\n",
      "Epoch 50/100\n",
      " - 8s - loss: 0.0838\n",
      "Epoch 51/100\n",
      " - 8s - loss: 0.0819\n",
      "Epoch 52/100\n",
      " - 8s - loss: 0.0801\n",
      "Epoch 53/100\n",
      " - 8s - loss: 0.0783\n",
      "Epoch 54/100\n",
      " - 8s - loss: 0.0766\n",
      "Epoch 55/100\n",
      " - 8s - loss: 0.0750\n",
      "Epoch 56/100\n",
      " - 8s - loss: 0.0734\n",
      "Epoch 57/100\n",
      " - 8s - loss: 0.0719\n",
      "Epoch 58/100\n",
      " - 8s - loss: 0.0703\n",
      "Epoch 59/100\n",
      " - 8s - loss: 0.0689\n",
      "Epoch 60/100\n",
      " - 9s - loss: 0.0675\n",
      "Epoch 61/100\n",
      " - 8s - loss: 0.0661\n",
      "Epoch 62/100\n",
      " - 8s - loss: 0.0648\n",
      "Epoch 63/100\n",
      " - 8s - loss: 0.0635\n",
      "Epoch 64/100\n",
      " - 8s - loss: 0.0622\n",
      "Epoch 65/100\n",
      " - 8s - loss: 0.0610\n",
      "Epoch 66/100\n",
      " - 9s - loss: 0.0598\n",
      "Epoch 67/100\n",
      " - 9s - loss: 0.0587\n",
      "Epoch 68/100\n",
      " - 8s - loss: 0.0575\n",
      "Epoch 69/100\n",
      " - 8s - loss: 0.0564\n",
      "Epoch 70/100\n",
      " - 8s - loss: 0.0553\n",
      "Epoch 71/100\n",
      " - 8s - loss: 0.0543\n",
      "Epoch 72/100\n",
      " - 8s - loss: 0.0533\n",
      "Epoch 73/100\n",
      " - 8s - loss: 0.0523\n",
      "Epoch 74/100\n",
      " - 8s - loss: 0.0513\n",
      "Epoch 75/100\n",
      " - 8s - loss: 0.0504\n",
      "Epoch 76/100\n",
      " - 8s - loss: 0.0494\n",
      "Epoch 77/100\n",
      " - 8s - loss: 0.0485\n",
      "Epoch 78/100\n",
      " - 8s - loss: 0.0477\n",
      "Epoch 79/100\n",
      " - 8s - loss: 0.0468\n",
      "Epoch 80/100\n",
      " - 8s - loss: 0.0460\n",
      "Epoch 81/100\n",
      " - 8s - loss: 0.0451\n",
      "Epoch 82/100\n",
      " - 8s - loss: 0.0443\n",
      "Epoch 83/100\n",
      " - 8s - loss: 0.0435\n",
      "Epoch 84/100\n",
      " - 8s - loss: 0.0427\n",
      "Epoch 85/100\n",
      " - 8s - loss: 0.0420\n",
      "Epoch 86/100\n",
      " - 8s - loss: 0.0413\n",
      "Epoch 87/100\n",
      " - 8s - loss: 0.0405\n",
      "Epoch 88/100\n",
      " - 8s - loss: 0.0398\n",
      "Epoch 89/100\n",
      " - 8s - loss: 0.0391\n",
      "Epoch 90/100\n",
      " - 8s - loss: 0.0384\n",
      "Epoch 91/100\n",
      " - 8s - loss: 0.0378\n",
      "Epoch 92/100\n",
      " - 8s - loss: 0.0371\n",
      "Epoch 93/100\n",
      " - 9s - loss: 0.0365\n",
      "Epoch 94/100\n",
      " - 8s - loss: 0.0358\n",
      "Epoch 95/100\n",
      " - 8s - loss: 0.0352\n",
      "Epoch 96/100\n",
      " - 8s - loss: 0.0346\n",
      "Epoch 97/100\n",
      " - 8s - loss: 0.0340\n",
      "Epoch 98/100\n",
      " - 9s - loss: 0.0334\n",
      "Epoch 99/100\n",
      " - 8s - loss: 0.0329\n",
      "Epoch 100/100\n",
      " - 8s - loss: 0.0323\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f13c942be10>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 2**15\n",
    "print('batch size:', batch_size)\n",
    "model.fit([lib_target, lib_context], labels, batch_size=batch_size, verbose=2, epochs=config['epochs'], validation_split=0.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.3 Extract our Embedding Matrix & Save to Disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T10:04:07.333588Z",
     "start_time": "2019-01-10T10:04:07.299114Z"
    }
   },
   "outputs": [],
   "source": [
    "x = embedding.get_weights()[0]\n",
    "y = lib_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the embedding to disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T10:04:08.164657Z",
     "start_time": "2019-01-10T10:04:08.161183Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'../datasets/javascript/models/library_vectors_dim100'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_prefix = rootDir + 'models/library_vectors_dim' + str(config['vector_dim'])\n",
    "file_prefix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T10:04:08.581382Z",
     "start_time": "2019-01-10T10:04:08.358932Z"
    }
   },
   "outputs": [],
   "source": [
    "save_array(file_prefix + '_x.vec', x)\n",
    "save_array(file_prefix + '_y.vec', y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T10:04:08.899755Z",
     "start_time": "2019-01-10T10:04:08.847711Z"
    }
   },
   "outputs": [],
   "source": [
    "x = load_array(file_prefix + '_x.vec')\n",
    "y = load_array(file_prefix + '_y.vec')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save HD vectors as JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T10:04:09.627998Z",
     "start_time": "2019-01-10T10:04:09.516292Z"
    }
   },
   "outputs": [],
   "source": [
    "vecHD = {y[i]: x[i].tolist() for i,_ in enumerate(x)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-10T10:04:13.616627Z",
     "start_time": "2019-01-10T10:04:09.679258Z"
    }
   },
   "outputs": [],
   "source": [
    "hd_file = rootDir + 'models/vectorsHD_dim{}.json'.format(config['vector_dim'])\n",
    "with open(hd_file, 'w') as f:\n",
    "    json.dump(vecHD, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "! gzip -f $hd_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.4 Convert embedding to Word2Vec format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "w2v_file = rootDir + 'models/w2v_dim{}.txt'.format(config['vector_dim'])\n",
    "with open(w2v_file, 'w') as f:\n",
    "    f.write('{} {}\\n'.format(x.shape[0], x.shape[1]));\n",
    "    for i in range(0, len(y)):\n",
    "        f.write('{} {}\\n'.format(y[i], str(x[i].tolist()).replace(',','').replace('[','').replace(']','')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gzip -f $w2v_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Done!\n",
    "\n",
    "Your vectors are available in directory $rootDir/models\n",
    "\n",
    "- The raw vectors are files `library_vectors_dim*_x.vec` (embedding) and `library_vectors_dim*_y.vec` (labels).\n",
    "- JSON-formatted vectors are availabile in the file `vectorsHD_dim*.json.gz`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DIR = ../datasets/javascript/models\n",
      "data_trn_dim100.json\t      library_vectors_dim100_y.vec   w2v_dim100.txt.gz\n",
      "lib_dict_dim100.json\t      reversed_lib_dict_dim100.json\n",
      "library_vectors_dim100_x.vec  vectorsHD_dim100.json.gz\n"
     ]
    }
   ],
   "source": [
    "! echo DIR = {rootDir}models\n",
    "! ls $rootDir/models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What has the model learned?\n",
    "\n",
    "- You can visualize your trained embeddings with the [Visualization.ipynb](Visualization.ipynb) notebook.\n",
    "- You can run nearest neighbour searches with the [Evaluation.ipynb](Evaluation.ipynb) notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
