{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QJb3sp4ZCF_O"
   },
   "source": [
    "# Feature Engineering\n",
    "\n",
    "Notebook supporting the [**Do we know our data, as good as we know our tools** talk](https://devoxxuk19.confinabox.com/talk/VEM-8021/Do_we_know_our_data_as_good_as_we_know_our_tools%3F) at [Devoxx UK 2019](http://twitter.com/@DevoxxUK).\n",
    "\n",
    "The contents of the notebook is inspired by many sources.\n",
    "\n",
    "\n",
    "### High-level steps covered:\n",
    "\n",
    "- Find hidden information\n",
    "  - feature extraction\n",
    "  - applying math/statistical functions [DEMO - WALKTHRU]\n",
    "  - apply physics functions\n",
    "- Deal with too much data\n",
    "  - dimensionality reduction\n",
    "  - feature selection [DEMO - WALKTHRU]\n",
    "- Statistical Inference \n",
    "\n",
    "\n",
    "### Resources\n",
    "\n",
    "- [Basic Feature Engineering With Time Series Data in Python](http://machinelearningmastery.com/basic-feature-engineering-time-series-data-python/)\n",
    "- [Zillow Prize - EDA, Data Cleaning & Feature Engineering](https://www.kaggle.com/lauracozma/eda-data-cleaning-feature-engineering)\n",
    "- [Feature-wise transformations](https://distill.pub/2018/feature-wise-transformations)\n",
    "- [tsfresh - tsfresh is used to to extract characteristics from time series](https://tsfresh.readthedocs.io/en/latest/text/introduction.html)\n",
    "- [featuretools - an open source python framework for automated feature engineering](https://github.com/featuretools/featuretools/)\n",
    "- [Synthetic features and outliers notebook](https://colab.research.google.com/notebooks/mlcc/synthetic_features_and_outliers.ipynb?utm_source=mlcc&utm_campaign=colab-external&utm_medium=referral&utm_content=syntheticfeatures-colab&hl=en#scrollTo=jnKgkN5fHbGy)\n",
    "- [Preprocessing data](https://scikit-learn.org/stable/modules/preprocessing.html)\n",
    "- **[Useful cheatsheets](https://github.com/neomatrix369/awesome-ai-ml-dl/blob/master/README-details.md#cheatsheets)**\n",
    "\n",
    "Please refer to the [Slides](http://bit.ly/do-we-know-our-data) for the step here after."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ask all the questions you should ask with regards to the domain and related domains or sub-domains.\n",
    "\n",
    "It is a good idea to know the **why** part of the action, why are we doing what we are doing with the data, see the [five whys](https://en.wikipedia.org/wiki/5_Whys).\n",
    "\n",
    "Some ideas (of course, please come up with your own as well):\n",
    "\n",
    "- To find hidden information\n",
    "- To extract the essence of the data which is representative of the rest\n",
    "of the data\n",
    "- Improve training efficiency: accuracy, speed, good use of resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C8uyqenw17x9"
   },
   "source": [
    "#### Load Your Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  boston_housing_dataset.zip\n",
      "  inflating: column.header           \n",
      "  inflating: housing-unclean.csv     \n",
      "  inflating: housing.csv             \n",
      "  inflating: housing.names           \n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "if [[ ! -s boston_housing_dataset.zip ]]; then\n",
    "    curl -O -L https://github.com/neomatrix369/awesome-ai-ml-dl/releases/download/v0.1/boston_housing_dataset.zip\n",
    "fi\n",
    "\n",
    "unzip -o boston_housing_dataset.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 967
    },
    "colab_type": "code",
    "id": "8o5QgSW519Ar",
    "outputId": "fad93ebd-1fda-4bba-a68b-5fd974fcef04"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Names and descriptions of the fields of the Boston Housing dataset can be found at\n",
      "https://github.com/jbrownlee/Datasets/blob/master/housing.names\n",
      "\n",
      "1. Title: Boston Housing Data\n",
      "\n",
      "2. Sources:\n",
      "   (a) Origin:  This dataset was taken from the StatLib library which is\n",
      "                maintained at Carnegie Mellon University.\n",
      "   (b) Creator:  Harrison, D. and Rubinfeld, D.L. 'Hedonic prices and the \n",
      "                 demand for clean air', J. Environ. Economics & Management,\n",
      "                 vol.5, 81-102, 1978.\n",
      "   (c) Date: July 7, 1993\n",
      "\n",
      "3. Past Usage:\n",
      "   -   Used in Belsley, Kuh & Welsch, 'Regression diagnostics ...', Wiley, \n",
      "       1980.   N.B. Various transformations are used in the table on\n",
      "       pages 244-261.\n",
      "    -  Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning.\n",
      "       In Proceedings on the Tenth International Conference of Machine \n",
      "       Learning, 236-243, University of Massachusetts, Amherst. Morgan\n",
      "       Kaufmann.\n",
      "\n",
      "4. Relevant Information:\n",
      "\n",
      "   Concerns housing values in suburbs of Boston.\n",
      "\n",
      "5. Number of Instances: 506\n",
      "\n",
      "6. Number of Attributes: 13 continuous attributes (including \"class\"\n",
      "                         attribute \"MEDV\"), 1 binary-valued attribute.\n",
      "\n",
      "7. Attribute Information:\n",
      "\n",
      "    1. CRIM      per capita crime rate by town\n",
      "    2. ZN        proportion of residential land zoned for lots over \n",
      "                 25,000 sq.ft.\n",
      "    3. INDUS     proportion of non-retail business acres per town\n",
      "    4. CHAS      Charles River dummy variable (= 1 if tract bounds \n",
      "                 river; 0 otherwise)\n",
      "    5. NOX       nitric oxides concentration (parts per 10 million)\n",
      "    6. RM        average number of rooms per dwelling\n",
      "    7. AGE       proportion of owner-occupied units built prior to 1940\n",
      "    8. DIS       weighted distances to five Boston employment centres\n",
      "    9. RAD       index of accessibility to radial highways\n",
      "    10. TAX      full-value property-tax rate per $10,000\n",
      "    11. PTRATIO  pupil-teacher ratio by town\n",
      "    12. B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks \n",
      "                 by town\n",
      "    13. LSTAT    % lower status of the population\n",
      "    14. MEDV     Median value of owner-occupied homes in $1000's\n",
      "\n",
      "8. Missing Attribute Values:  None.\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "sns.set()\n",
    "\n",
    "names = [line.strip() for line in open(\"column.header\", 'r')]\n",
    "data = pd.read_csv(\"housing.csv\", names=names)\n",
    "\n",
    "print(\"Names and descriptions of the fields of the Boston Housing dataset can be found at\")\n",
    "print(\"https://github.com/jbrownlee/Datasets/blob/master/housing.names\")\n",
    "print(\"\")\n",
    "!cat housing.names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's set a baseline to evaluate our attempts to improve our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7261570836552489"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "def evaluateModelOnDataset(data):\n",
    "    X_train, X_test, y_train, y_test = train_test_split(data.loc[:, data.columns != 'medv'].values, data[\"medv\"].values, test_size=0.33, random_state=42)\n",
    "    model = LinearRegression()\n",
    "    model.fit(X_train,y_train)\n",
    "    return model.score(X_test,y_test)\n",
    "\n",
    "def evaluateModelOnTrainingAndTestSets(X_train, X_test, y_train, y_test):\n",
    "    model = LinearRegression()\n",
    "    model.fit(X_train,y_train)\n",
    "    return model.score(X_test,y_test)\n",
    "evaluateModelOnDataset(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qIhcY8dYE6QZ"
   },
   "source": [
    "### Find hidden information\n",
    "\n",
    "- feature extraction\n",
    "- applying math/statistical functions\n",
    "- apply physics functions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OH3OVpoBJRtl"
   },
   "source": [
    "#### Feature extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Mm5EsDitJXgQ"
   },
   "source": [
    "- group together sparse classes\n",
    "- create new calculated columns, for e.g. extracting weekday from date\n",
    "- generate relevant labels with the help of results from clustering\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Mm5EsDitJXgQ"
   },
   "source": [
    "#### Applying math/statistical functions [DEMO - WALKTHRU]\n",
    "\n",
    "- convert to absolute values\n",
    "- apply root mean square\n",
    "- use logarithmic functions\n",
    "- applying rolling mean / stddev / min / max \n",
    "\n",
    "And manage precision of the data!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observed earlier than there was a correlation between lstat and medv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "D8ZA1ghXGNAS"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 18164 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(data[\"lstat\"], data[\"medv\"], marker='o')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">when we plot lstat (used -lstat in the second plot to make it more obvious) in function of medv, we see an **exponential curve**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 18236 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(-data[\"lstat\"], data[\"medv\"], marker='o')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">We apply **logarithm function** to it to convert it into a linear correlation and make it easier for our model to pick the correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 18800 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_engineered=data.copy()\n",
    "data_engineered[\"lstat\"]=np.log(data_engineered[\"lstat\"])\n",
    "plt.scatter(data_engineered[\"lstat\"], data_engineered[\"medv\"], marker='o')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "previous accuracy: 0.726157\n",
      "new accuracy: 0.772251\n"
     ]
    }
   ],
   "source": [
    "print(\"previous accuracy: %f\" % evaluateModelOnDataset(data))\n",
    "print(\"new accuracy: %f\" % evaluateModelOnDataset(data_engineered))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">we can gain more than 4% in accuracy with this transformation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SEPARsYEJrnn"
   },
   "source": [
    "#### Apply physics functions\n",
    "- Energy\n",
    "- Energy rate\n",
    "- Short Term Average / Long term Avg\n",
    "- Kurtosis\n",
    "- FFT (Fast Fourier Transform)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SEPARsYEJrnn"
   },
   "source": [
    "#### Feature scaling\n",
    "- Standardization\n",
    "- Normalization\n",
    "- map to uniform / gaussian distribution\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data.loc[:, data.columns != 'medv'].values, data[\"medv\"].values, test_size=0.33, random_state=42)\n",
    "X_train_normalized=preprocessing.normalize(X_train)\n",
    "X_test_normalized=preprocessing.normalize(X_test)\n",
    "X_train_standardized=preprocessing.scale(X_train)\n",
    "X_test_standardized=preprocessing.scale(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline accuracy: 0.726157\n",
      "new accuracy for standardized data: 0.704179\n",
      "new accuracy for normalized data: 0.739282\n"
     ]
    }
   ],
   "source": [
    "print(\"baseline accuracy: %f\" % evaluateModelOnTrainingAndTestSets(X_train, X_test, y_train, y_test))\n",
    "print(\"new accuracy for standardized data: %f\" % evaluateModelOnTrainingAndTestSets(X_train_standardized, X_test_standardized, y_train, y_test))\n",
    "print(\"new accuracy for normalized data: %f\" % evaluateModelOnTrainingAndTestSets(X_train_normalized, X_test_normalized, y_train, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">We can gain 1% in accuracy with normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dSdqmARsDwbM"
   },
   "source": [
    "### Deal with too many features / too much data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4QO_AyToFPBw"
   },
   "source": [
    "- dimensionality reduction\n",
    "- feature selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kP4yEElUGNhi"
   },
   "source": [
    "#### Dimensionality reduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Factorisation \n",
    "    - PCA Principal Component Analysis\n",
    "- ICA \n",
    "Independent Component Analysis\n",
    "- t-SNE \n",
    "t-Distributed Stochastic Neighbour Embedding\n",
    "- UMAP \n",
    "Uniform Manifold Approximation and Projection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ck3LgHNHKCf5"
   },
   "source": [
    "#### Correlation Between Attributes\n",
    "\n",
    "Correlation refers to the relationship between two variables and how they may or may not change together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R5bFngNdFYWF"
   },
   "source": [
    "#### Feature selection [DEMO - WALKTHRU]\n",
    "\n",
    "Manual\n",
    "- Filter out features which are highly correlated.\n",
    "- Plot multi scatter chart \n",
    "- Use correlation table \n",
    "- (might need to remove features which were used for extraction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> given that it's a small dataset, we are not going to yield much better results by removing too many features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1014
    },
    "colab_type": "code",
    "id": "Ps3O65S9hzwn",
    "outputId": "4dfb4962-6f30-43c2-e6cb-483aa365535b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          crim     zn  indus   chas    nox     rm    age    dis    rad    tax  ptratio      b  \\\n",
      "crim     1.000 -0.200  0.407 -0.056  0.421 -0.219  0.353 -0.380  0.626  0.583    0.290 -0.385   \n",
      "zn      -0.200  1.000 -0.534 -0.043 -0.517  0.312 -0.570  0.664 -0.312 -0.315   -0.392  0.176   \n",
      "indus    0.407 -0.534  1.000  0.063  0.764 -0.392  0.645 -0.708  0.595  0.721    0.383 -0.357   \n",
      "chas    -0.056 -0.043  0.063  1.000  0.091  0.091  0.087 -0.099 -0.007 -0.036   -0.122  0.049   \n",
      "nox      0.421 -0.517  0.764  0.091  1.000 -0.302  0.731 -0.769  0.611  0.668    0.189 -0.380   \n",
      "rm      -0.219  0.312 -0.392  0.091 -0.302  1.000 -0.240  0.205 -0.210 -0.292   -0.356  0.128   \n",
      "age      0.353 -0.570  0.645  0.087  0.731 -0.240  1.000 -0.748  0.456  0.506    0.262 -0.274   \n",
      "dis     -0.380  0.664 -0.708 -0.099 -0.769  0.205 -0.748  1.000 -0.495 -0.534   -0.232  0.292   \n",
      "rad      0.626 -0.312  0.595 -0.007  0.611 -0.210  0.456 -0.495  1.000  0.910    0.465 -0.444   \n",
      "tax      0.583 -0.315  0.721 -0.036  0.668 -0.292  0.506 -0.534  0.910  1.000    0.461 -0.442   \n",
      "ptratio  0.290 -0.392  0.383 -0.122  0.189 -0.356  0.262 -0.232  0.465  0.461    1.000 -0.177   \n",
      "b       -0.385  0.176 -0.357  0.049 -0.380  0.128 -0.274  0.292 -0.444 -0.442   -0.177  1.000   \n",
      "lstat    0.456 -0.413  0.604 -0.054  0.591 -0.614  0.602 -0.497  0.489  0.544    0.374 -0.366   \n",
      "medv    -0.388  0.360 -0.484  0.175 -0.427  0.695 -0.377  0.250 -0.382 -0.469   -0.508  0.333   \n",
      "\n",
      "         lstat   medv  \n",
      "crim     0.456 -0.388  \n",
      "zn      -0.413  0.360  \n",
      "indus    0.604 -0.484  \n",
      "chas    -0.054  0.175  \n",
      "nox      0.591 -0.427  \n",
      "rm      -0.614  0.695  \n",
      "age      0.602 -0.377  \n",
      "dis     -0.497  0.250  \n",
      "rad      0.489 -0.382  \n",
      "tax      0.544 -0.469  \n",
      "ptratio  0.374 -0.508  \n",
      "b       -0.366  0.333  \n",
      "lstat    1.000 -0.738  \n",
      "medv    -0.738  1.000  \n",
      "\n",
      "A correlation of -1 or 1 shows a full negative (inverse) or positive (direct) correlation respectively.\n",
      "A value of 0 shows no correlation at all.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fe4c6729a20>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 81932 characters]\n",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Pairwise Pearson correlations\n",
    "pd.set_option('display.width', 100)\n",
    "pd.set_option('precision', 3)\n",
    "correlations = data.corr(method='pearson')\n",
    "print(correlations)\n",
    "print()\n",
    "print(\"A correlation of -1 or 1 shows a full negative (inverse) or positive (direct) correlation respectively.\") \n",
    "print(\"A value of 0 shows no correlation at all.\")\n",
    "\n",
    "# Generate a mask for the upper triangle\n",
    "mask = np.zeros_like(correlations, dtype=np.bool)\n",
    "mask[np.triu_indices_from(mask)] = True\n",
    "\n",
    "# Set up the matplotlib figure\n",
    "f, ax = plt.subplots(figsize=(10, 10));\n",
    "\n",
    "# Generate a custom diverging colormap\n",
    "cmap = sns.diverging_palette(220, 10, as_cmap=True)\n",
    "\n",
    "# Draw the heatmap with the mask and correct aspect ratio\n",
    "sns.heatmap(correlations, mask=mask, cmap=cmap, vmax=.3, center=0,\n",
    "            square=True, linewidths=.5, cbar_kws={\"shrink\": .5},annot=True)\n",
    "# correlation_matrix = boston.corr().round(2)\n",
    "# annot = True to print the values inside the square\n",
    "# sns.heatmap(correlations, annot=True,mask=mask, cmap=cmap)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">The features with high correlations between themselves may lower performances if used together in case of some models (linear regression): \n",
    "- tax and rad (0.91)\n",
    "- dis and nox (-0.77)\n",
    "- dis and indus (-0.71)\n",
    "- dis and age (-0.75)\n",
    "\n",
    ">No feature seems to show any absence of correlation with the target feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VTwnUrMVLBss"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline: 0.726157\n",
      "accuracy without crim: 0.724202\n",
      "accuracy without zn: 0.714156\n",
      "accuracy without indus: 0.727770            - improvement\n",
      "accuracy without chas: 0.725516\n",
      "accuracy without nox: 0.708281\n",
      "accuracy without rm: 0.686267\n",
      "accuracy without age: 0.729383            - improvement\n",
      "accuracy without dis: 0.689851\n",
      "accuracy without rad: 0.704817\n",
      "accuracy without tax: 0.714122\n",
      "accuracy without ptratio: 0.693525\n",
      "accuracy without b: 0.729646            - improvement\n",
      "accuracy without lstat: 0.687862\n"
     ]
    }
   ],
   "source": [
    "prev_accuracy =evaluateModelOnDataset(data)\n",
    "print(\"baseline: %f\" % prev_accuracy)\n",
    "for feature in data.columns:\n",
    "    if feature!=\"medv\":\n",
    "        new_accuracy=evaluateModelOnDataset(data.loc[:, data.columns != feature])\n",
    "        if new_accuracy > prev_accuracy:\n",
    "            print(\"accuracy without %s: %f            - improvement\" % (feature, new_accuracy))\n",
    "        else: \n",
    "            print(\"accuracy without %s: %f\" % (feature, new_accuracy))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7336244309845095"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluateModelOnDataset(data.loc[:, (data.columns != \"age\")&(data.columns != \"indus\")&(data.columns != \"b\")])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Results are not obvious, since probably we have a small dataset. \n",
    "\n",
    ">But if we remove age and indus which are highly correlated with dis, we see a small improvement. \n",
    "We can gain less than 1% in accuracy by removing a couple of features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R5bFngNdFYWF"
   },
   "source": [
    "Programmatically\n",
    "- Tree based feature selection\n",
    "- Use feature importance from XGBoost or RandomForest\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy of randomForest on full dataset: 0.975159\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jeremie/.local/lib/python3.6/site-packages/sklearn/ensemble/forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 10428 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "\n",
    "labels=data.loc[:, data.columns != 'medv'].columns.values\n",
    "\n",
    "model = RandomForestRegressor();\n",
    "model.fit(data[labels],data[\"medv\"].values);\n",
    "print(\"accuracy of randomForest on full dataset: %f\" % model.score(data[labels],data[\"medv\"].values));\n",
    "\n",
    "df_features=pd.DataFrame()\n",
    "df_features[\"feature\"]=labels\n",
    "df_features[\"importance\"]=model.feature_importances_\n",
    "df_features.sort_values(by=\"importance\",inplace=True)\n",
    "\n",
    "y_pos = np.arange(len(df_features))\n",
    "plt.barh(y_pos,df_features[\"importance\"])\n",
    "plt.yticks(y_pos,df_features[\"feature\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline: 0.726157\n",
      "with top 3 features: 0.624183\n",
      "with last 2 features removed: 0.714106\n"
     ]
    }
   ],
   "source": [
    "print(\"baseline: %f\" % evaluateModelOnDataset(data[[\"crim\",\"zn\",\"indus\",\"chas\",\"nox\",\"rm\",\"age\",\"dis\",\"rad\",\"tax\",\"ptratio\",\"b\",\"lstat\",\"medv\"]]))\n",
    "print(\"with top 3 features: %f\" % evaluateModelOnDataset(data[[\"lstat\",\"rm\",\"dis\",\"medv\"]]))\n",
    "print(\"with last 2 features removed: %f\" % evaluateModelOnDataset(data.loc[:, (data.columns != 'chas')&(data.columns != 'zn')]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">We don't get better results by picking the top or removing the last features from Random Forest. It can be explained by the size of the dataset which doesn't require reducing its number of features. \n",
    "\n",
    ">We do observe though that the features we removed earlier (indus, b and age) are among the least important features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5OisyWxwGtCa"
   },
   "source": [
    "### Statistical Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QbsxTd1zGwsh"
   },
   "source": [
    "- [Understanding statistical inference]() [video]\n",
    "- [Four ideas of Statistical Inference](http://www.bristol.ac.uk/medical-school/media/rms/red/4_ideas_of_statistical_inference.html)\n",
    "- [An Introduction to Statistical Learning](http://www-bcf.usc.edu/~gareth/ISL/) [book]\n",
    "- [Statistical Inference](https://www.coursera.org/learn/statistical-inference) [course]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparatory questions to ask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Can I create a model from my dataset efficiently given my resources?\n",
    "- How do we make best use of the trial-error process yet get the best out of the dataset / model?\n",
    "- How can I make informed decisions and record them and reuse them in my next iteration?\n",
    "- How can I make sure that the end model is useful to solve my business problem after going through the tedious process?\n",
    "- From the trends we identified previously can we make it simpler for the model to pick and use such information from the dataset?\n",
    "- How can I get the essence out of my original dataset ? Can I express the intuitions I got during the analysis in a more obvious way ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5ZVFw4kaHVtO"
   },
   "source": [
    "### Please refer to the [Slides](http://bit.ly/do-you-know-your-data) for the step here after."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Feature Engineering (Do we know our data).ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
