{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.10"
    },
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "23tJs1SNuGuu",
        "ozfCAd5-uGuv"
      ],
      "toc_visible": true,
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Sahel-Eskandar/Project-Notebooks/blob/main/8_Tulog.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CCevKu5BuGuo"
      },
      "source": [
        "# Orion Tulog\n",
        "\n",
        "This is a tutorial for unsupervised anomaly detection in time series using Orion. Orion is a `python` package for time series anoamly detection. It provides a suite of both statistical and machine learning models that enable efficient anomaly detection.\n",
        "\n",
        "In this tutorial, we will learn how to set up Orion, train a machine learning model, and perform anomaly detection. We will delve into each part seperately and then run the evaluation pipeline from beginning to end in order to compare multiple models against each other."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BYo47twWuHdV"
      },
      "source": [
        "! pip install orion-ml 'urllib3>=1.25.4,<1.26'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kP7lK5Vldphk"
      },
      "source": [
        "### Remember to restart runtime!\n",
        "\n",
        "Let's view the installed Orion version"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oI1pEHHZdk4b",
        "outputId": "9ee2afef-d991-455e-a303-023238a87446"
      },
      "source": [
        "! pip freeze | grep orion-ml"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "orion-ml==0.1.6\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NLM45S-Khz4z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "06cfcb1a-d41a-4ec1-c6bb-27718855578c"
      },
      "source": [
        "!pip install utils"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting utils\n",
            "  Downloading https://files.pythonhosted.org/packages/55/e6/c2d2b2703e7debc8b501caae0e6f7ead148fd0faa3c8131292a599930029/utils-1.0.1-py2.py3-none-any.whl\n",
            "Installing collected packages: utils\n",
            "Successfully installed utils-1.0.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ub-WCxf6ijLv",
        "outputId": "7ea559e4-5753-4673-828b-2bd600659f7c"
      },
      "source": [
        "!pip install matplotlib"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (3.2.2)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.4.7)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (0.10.0)\n",
            "Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.16.6)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.8.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from cycler>=0.10->matplotlib) (1.15.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KZmgNi27erXb"
      },
      "source": [
        "To properly run the Tulog, we require some supporting files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7zLKYyQwVH2u",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7524ffc5-10b8-4991-db03-4ae0185e4748"
      },
      "source": [
        "%%bash\n",
        "rm -rf Orion\n",
        "rm -rf images\n",
        "\n",
        "git clone https://github.com/signals-dev/Orion.git\n",
        "mv Orion/notebooks/tulog/* .\n",
        "exit"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'Orion'...\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rozmsLq1hulA"
      },
      "source": [
        "from utils import plot, plot_ts, plot_rws, plot_error, unroll_ts"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MZ2mtAcBuGup"
      },
      "source": [
        "# general imports \n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "from sklearn.impute import SimpleImputer\n",
        "from sklearn.preprocessing import MinMaxScaler"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g7ErOxIluGuq"
      },
      "source": [
        "## Part 1\n",
        "In part one of the series, we explore a time series data, particularly the NYC taxi data. You can find the raw data on the [TLC](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page) or the processed version maintained by Numenta [here](https://github.com/numenta/NAB/tree/master/data). We also explore what reasons could possibily be contributing to producing such anomalies.\n",
        "\n",
        "### Data Loading\n",
        "There is a collection of data already available in Orion, to load them, we use the `load_signal` function and pass the name of the signal we wish to obtain. Similarly, since this data is labeled, we use the `load_anomalies` function to get the corresponding anomaly of the signal"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c8pBPQhsuGuq"
      },
      "source": [
        "from orion.data import load_signal, load_anomalies"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2eIRwDy-uGur",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "4409374a-125d-47a4-d864-52a0e1238f1f"
      },
      "source": [
        "signal = 'nyc_taxi'\n",
        "\n",
        "# load signal\n",
        "df = load_signal(signal)\n",
        "\n",
        "# load ground truth anomalies\n",
        "known_anomalies = load_anomalies(signal)\n",
        "\n",
        "df.head(5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>timestamp</th>\n",
              "      <th>value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1404165600</td>\n",
              "      <td>10844.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1404167400</td>\n",
              "      <td>8127.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1404169200</td>\n",
              "      <td>6210.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1404171000</td>\n",
              "      <td>4656.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1404172800</td>\n",
              "      <td>3820.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    timestamp    value\n",
              "0  1404165600  10844.0\n",
              "1  1404167400   8127.0\n",
              "2  1404169200   6210.0\n",
              "3  1404171000   4656.0\n",
              "4  1404172800   3820.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AxH77hfEuGur",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 263
        },
        "outputId": "fa39ae90-0ef0-4ece-c623-16bcac2ff5b8"
      },
      "source": [
        "plot(df, known_anomalies)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "[encoded data, 290748 characters]\n",
            "text/plain": [
              "<Figure size 2160x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0KjpmaPouGur"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n7YhH1UouGus"
      },
      "source": [
        "## Part 2\n",
        "\n",
        "In part two of the series, we look at anomaly detection through time series reconstruction, particularly using a GAN model. We go through a sequence of transformations and data preparation, as well as model training and prediction.\n",
        "\n",
        "### Orion API\n",
        "We will use `Orion` to perform these sequence of actions. We will be emphasizing the usage of the `TadGAN` model which is a time series anomaly detection using GANs model. The model is specified in a `json` format accompanied with this notebook named `tadgan.json`. There are more pipelines defined within the repository including: ARIMA, LSTM, etc. \n",
        "\n",
        "The Orion API is a simple interface that allows you to interact with anomaly detection pipeline. To train the model on the data, we simply use the `fit` method; to do anomaly detection, we use the `detect` method. In our case, we want to fit the data and then perform detection; therefore we use the `fit_detect` method. This might take some time to run. Once it’s done, we can visualize the results.\n",
        "\n",
        "**Note**: the model might take some time to train. For experimentation purposes, you can reduce the number of `epochs` in the `tadgan.json` file such that you reduce the number of training iterations. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jy9ESebbuGus",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "61febf7f-bd3b-47c7-cee1-af3d0f26b83f"
      },
      "source": [
        "from orion import Orion\n",
        "\n",
        "\n",
        "orion = Orion(\n",
        "    pipeline='tadgan.json'\n",
        ")\n",
        "\n",
        "anomalies = orion.fit_detect(df)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "If using Keras pass *_constraint arguments to layers.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py:297: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set `model.trainable` without calling `model.compile` after ?\n",
            "  'Discrepancy between trainable weights and collected trainable'\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py:297: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set `model.trainable` without calling `model.compile` after ?\n",
            "  'Discrepancy between trainable weights and collected trainable'\n",
            "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py:297: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set `model.trainable` without calling `model.compile` after ?\n",
            "  'Discrepancy between trainable weights and collected trainable'\n",
            "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py:297: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set `model.trainable` without calling `model.compile` after ?\n",
            "  'Discrepancy between trainable weights and collected trainable'\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 1/35, [Dx loss: [-0.59666723 -3.0611277   1.623648    0.08408127]] [Dz loss: [-1.2433692  -1.0134093  -1.6093578   0.13793984]] [G loss: [ 2.6131501  -1.6092415   2.6970034   0.15253882]]\n",
            "Epoch: 2/35, [Dx loss: [-0.84876996 -4.8582134   3.84868     0.01607621]] [Dz loss: [-2.9175756  -1.361662   -2.8721087   0.13161944]] [G loss: [ 1.2221725  -3.7823207   3.6977804   0.13067132]]\n",
            "Epoch: 3/35, [Dx loss: [-0.62313867 -8.245832    7.446357    0.01763361]] [Dz loss: [-1.1950797  -0.9493746  -0.7540207   0.05083155]] [G loss: [-5.321727   -7.427534    1.0115331   0.10942739]]\n",
            "Epoch: 4/35, [Dx loss: [-0.07653887 -5.15995     4.9135303   0.01698867]] [Dz loss: [-0.7568386  -0.7212786  -0.42734724  0.0391787 ]] [G loss: [-2.9785624  -4.6830425   0.65672505  0.10477553]]\n",
            "Epoch: 5/35, [Dx loss: [ -0.27141     20.914242   -21.409687     0.02240377]] [Dz loss: [-1.2125322  -0.44324878 -1.201042    0.04317584]] [G loss: [24.46362    21.994528    1.5388612   0.09302299]]\n",
            "Epoch: 6/35, [Dx loss: [-0.53530407  0.06765649 -0.70267886  0.00997191]] [Dz loss: [-0.9653922   0.15390505 -1.4802991   0.03610016]] [G loss: [2.7415652  0.19406657 1.8647192  0.06827794]]\n",
            "Epoch: 7/35, [Dx loss: [-5.0845575e-01 -1.1954759e+01  1.1335342e+01  1.1095214e-02]] [Dz loss: [-1.2547866   0.33047146 -1.9726511   0.0387393 ]] [G loss: [ -7.2485614  -10.376204     2.445203     0.06824415]]\n",
            "Epoch: 8/35, [Dx loss: [-5.0946206e-01  2.0655043e+01 -2.1288086e+01  1.2358168e-02]] [Dz loss: [-1.2367618   0.2843241  -1.9343969   0.04133104]] [G loss: [23.870205   20.878721    2.4091039   0.05823804]]\n",
            "Epoch: 9/35, [Dx loss: [-0.21517307  7.820976   -8.240185    0.02040382]] [Dz loss: [-1.5542487   3.658452   -6.1761293   0.09634285]] [G loss: [16.893532    8.487561    7.559762    0.08462046]]\n",
            "Epoch: 10/35, [Dx loss: [-0.51729053  7.8816295  -8.498292    0.00993708]] [Dz loss: [ 0.45849162  1.9898001  -1.8830092   0.03517003]] [G loss: [10.857022   8.339752   2.0415802  0.0475691]]\n",
            "Epoch: 11/35, [Dx loss: [-4.2986190e-01  1.0998393e+01 -1.1534571e+01  1.0631477e-02]] [Dz loss: [ 0.3399183   0.5875884  -0.8070243   0.05593541]] [G loss: [13.4672165  11.850769    1.0919731   0.05244743]]\n",
            "Epoch: 12/35, [Dx loss: [-2.0283067e-01  2.7495499e+01 -2.7781322e+01  8.2990434e-03]] [Dz loss: [-0.39967144  0.21876056 -1.0256109   0.0407179 ]] [G loss: [30.008917   28.19517     1.3427088   0.04710359]]\n",
            "Epoch: 13/35, [Dx loss: [-5.1400596e-01  2.9920559e+01 -3.0550478e+01  1.1591732e-02]] [Dz loss: [-0.974045    3.1353586  -5.29678     0.11873762]] [G loss: [37.90646    30.323357    6.69121     0.08918878]]\n",
            "Epoch: 14/35, [Dx loss: [-3.2433611e-01  1.7999472e+01 -1.8435814e+01  1.1201040e-02]] [Dz loss: [ 1.3514206   2.4456992  -1.4870478   0.03927689]] [G loss: [20.400528   18.424484    1.5036849   0.04723642]]\n",
            "Epoch: 15/35, [Dx loss: [-2.6881868e-01  1.9213297e+01 -1.9579308e+01  9.7185792e-03]] [Dz loss: [ 0.76408345  1.0789229  -0.6965198   0.03816803]] [G loss: [20.92587    19.625095    0.81698287  0.04837957]]\n",
            "Epoch: 16/35, [Dx loss: [-2.8212649e-01  1.6598167e+01 -1.6972050e+01  9.1748862e-03]] [Dz loss: [ 0.35429046  0.68219036 -0.7424168   0.04145165]] [G loss: [18.228775   16.939074    0.85323966  0.04364655]]\n",
            "Epoch: 17/35, [Dx loss: [-4.0103215e-01  1.4907794e+01 -1.5388024e+01  7.9206126e-03]] [Dz loss: [-1.7914994  -0.6995772  -1.651058    0.05591361]] [G loss: [17.783094   15.4120655   1.9343946   0.04366355]]\n",
            "Epoch: 18/35, [Dx loss: [-1.0849687e-01  1.4302131e+01 -1.4489874e+01  7.9248166e-03]] [Dz loss: [-0.3008776  -0.13101429 -0.5016261   0.03317628]] [G loss: [15.679148   14.561571    0.7232206   0.03943587]]\n",
            "Epoch: 19/35, [Dx loss: [-3.5651132e-01  2.0787487e+01 -2.1217600e+01  7.3598279e-03]] [Dz loss: [-1.4908726  -0.81511104 -1.0471147   0.03713529]] [G loss: [22.752956   21.272585    1.1508065   0.03295672]]\n",
            "Epoch: 20/35, [Dx loss: [-2.0807849e-01  2.2560335e+01 -2.2838511e+01  7.0099216e-03]] [Dz loss: [-1.5422878  -1.3601342  -0.58832854  0.04061742]] [G loss: [24.109577   22.901133    0.8514659   0.03569752]]\n",
            "Epoch: 21/35, [Dx loss: [-2.6032192e-01  1.7759491e+01 -1.8083344e+01  6.3524139e-03]] [Dz loss: [-0.6631293  -0.03375707 -0.89810663  0.02687347]] [G loss: [19.345419   18.05504     0.9793737   0.03110044]]\n",
            "Epoch: 22/35, [Dx loss: [-2.7508944e-01  2.1647430e+01 -2.1991156e+01  6.8633864e-03]] [Dz loss: [-1.8164939  -1.7594303  -0.40072033  0.03436568]] [G loss: [23.137072   22.24874     0.5302737   0.03580608]]\n",
            "Epoch: 23/35, [Dx loss: [-2.4496344e-01  2.4776434e+01 -2.5103552e+01  8.2157627e-03]] [Dz loss: [-1.0389194  -2.0121243   0.7324158   0.02407903]] [G loss: [24.64422    25.01356    -0.6481564   0.02788173]]\n",
            "Epoch: 24/35, [Dx loss: [-1.6627362e-01  2.8517323e+01 -2.8756886e+01  7.3296614e-03]] [Dz loss: [-1.0068874  -1.9418709   0.7021407   0.02328426]] [G loss: [28.707392   28.986391   -0.65873885  0.03797436]]\n",
            "Epoch: 25/35, [Dx loss: [-2.4259941e-01  3.4827797e+01 -3.5154305e+01  8.3919428e-03]] [Dz loss: [-1.3306577  -1.1377838  -0.45057333  0.02576992]] [G loss: [3.6067711e+01 3.5142281e+01 5.6876528e-01 3.5666935e-02]]\n",
            "Epoch: 26/35, [Dx loss: [-1.9545266e-01  2.8685261e+01 -2.8968990e+01  8.8281725e-03]] [Dz loss: [-1.1780816  -0.6078882  -0.81831425  0.02481204]] [G loss: [29.821716   28.633595    0.85472876  0.03333925]]\n",
            "Epoch: 27/35, [Dx loss: [-2.5898948e-01  3.2807938e+01 -3.3154964e+01  8.8047469e-03]] [Dz loss: [-1.055471   -1.7681121   0.49685776  0.02157834]] [G loss: [33.427925   33.586304   -0.5067116   0.03483275]]\n",
            "Epoch: 28/35, [Dx loss: [-1.8336746e-01  3.6067036e+01 -3.6333977e+01  8.3587952e-03]] [Dz loss: [-1.3581749  -3.2526386   1.619181    0.02752821]] [G loss: [ 3.4899876e+01  3.6179806e+01 -1.6089212e+00  3.2898672e-02]]\n",
            "Epoch: 29/35, [Dx loss: [-1.6511586e-01  4.1235035e+01 -4.1481220e+01  8.1068277e-03]] [Dz loss: [-1.3578347  -3.3297083   1.7071255   0.02647476]] [G loss: [ 4.0362999e+01  4.1654339e+01 -1.6434319e+00  3.5208616e-02]]\n",
            "Epoch: 30/35, [Dx loss: [-2.4463809e-01  3.4779083e+01 -3.5098953e+01  7.5231837e-03]] [Dz loss: [-1.3139405  -3.535485    1.9449053   0.02766392]] [G loss: [ 3.3597389e+01  3.5198612e+01 -1.9048164e+00  3.0359808e-02]]\n",
            "Epoch: 31/35, [Dx loss: [-1.4440717e-01  4.7327011e+01 -4.7562263e+01  9.0871453e-03]] [Dz loss: [-1.6801788  -2.8940096   0.9113296   0.03025012]] [G loss: [ 4.7370487e+01  4.7851570e+01 -8.4915918e-01  3.6808651e-02]]\n",
            "Epoch: 32/35, [Dx loss: [-3.2612553e-01  4.7148296e+01 -4.7565994e+01  9.1578355e-03]] [Dz loss: [-1.4717331  -2.8822389   1.1486092   0.02618968]] [G loss: [ 4.6834145e+01  4.7624805e+01 -1.1167628e+00  3.2609995e-02]]\n",
            "Epoch: 33/35, [Dx loss: [-1.8641123e-01  6.6178581e+01 -6.6456551e+01  9.1572311e-03]] [Dz loss: [-1.6306453  -2.2815797   0.36867186  0.02822633]] [G loss: [ 6.6748192e+01  6.6681602e+01 -2.8900075e-01  3.5557918e-02]]\n",
            "Epoch: 34/35, [Dx loss: [-1.1144198e-01  4.3896709e+01 -4.4089809e+01  8.1645409e-03]] [Dz loss: [-1.4746797 -2.4014604  0.6592723  0.0267509]] [G loss: [ 4.3453106e+01  4.3731434e+01 -6.2115872e-01  3.4282811e-02]]\n",
            "Epoch: 35/35, [Dx loss: [-2.6848543e-01  4.1562862e+01 -4.1911343e+01  7.9996828e-03]] [Dz loss: [-1.4039247 -2.7740717  1.0908946  0.0279253]] [G loss: [ 4.1059177e+01  4.1785114e+01 -1.0636271e+00  3.3769589e-02]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N5DBgXfguGus"
      },
      "source": [
        "Let's visualize the results."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lDrU6BGLuGut",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 450
        },
        "outputId": "097076a0-2033-4ce0-b307-b4f7e11cbcad"
      },
      "source": [
        "plot(df, [anomalies, known_anomalies])\n",
        "anomalies.head(5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "[encoded data, 291772 characters]\n",
            "text/plain": [
              "<Figure size 2160x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>start</th>\n",
              "      <th>end</th>\n",
              "      <th>severity</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1404387000</td>\n",
              "      <td>1404729000</td>\n",
              "      <td>0.606416</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1410355800</td>\n",
              "      <td>1410625800</td>\n",
              "      <td>0.252006</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1413428400</td>\n",
              "      <td>1413610200</td>\n",
              "      <td>0.026142</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1414620000</td>\n",
              "      <td>1414953000</td>\n",
              "      <td>0.284177</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1422284400</td>\n",
              "      <td>1422577800</td>\n",
              "      <td>0.343403</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        start         end  severity\n",
              "0  1404387000  1404729000  0.606416\n",
              "1  1410355800  1410625800  0.252006\n",
              "2  1413428400  1413610200  0.026142\n",
              "3  1414620000  1414953000  0.284177\n",
              "4  1422284400  1422577800  0.343403"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zfe79jHXuGut"
      },
      "source": [
        "The red intervals depict the detected anomalies, the green intervals show the ground truth. Cool! the model was able to detect some anomalies. We also see that it detected some other intervals that were not included in the ground truth labels. It is clear though, they are falling out of shape with respect to the remaining signal.\n",
        "> **note**: the results might differ between runs.\n",
        "\n",
        "\n",
        "\n",
        "We might have jumped straight to the results but let's trace back and look at what the model actually did.\n",
        "\n",
        "There is a series of transformations happening to the data in order to obtain the result you have just seen. From data preprocessing, model training, to post-processing functionalities. We specify these functions, which we refer to as primitives, within the model’s `.json` file. What are these primitives? If we were to look at the `tadgan.json` model, we find these sequential primitives:\n",
        "\n",
        "```\n",
        "\"primitives\": [ \n",
        "    \"mlprimitives.custom.timeseries_preprocessing.time_segments_aggregate”,\n",
        "    \"sklearn.impute.SimpleImputer\",\n",
        "    \"sklearn.preprocessing.MinMaxScaler\",\n",
        "    \"mlprimitives.custom.timeseries_preprocessing.rolling_window_sequences\",\n",
        "    \"orion.primitives.tadgan.TadGAN\",\n",
        "    \"orion.primitives.tadgan.score_anomalies\",\n",
        "    \"orion.primitives.timeseries_anomalies.find_anomalies\"\n",
        "]\n",
        "```\n",
        "\n",
        "Each primitive is responsible for a single task. We describe the procedure of each primitive in the remainder of this notebook"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tnhXG9wiuGut"
      },
      "source": [
        "### Data Preparation \n",
        "#### A. data frequency\n",
        "Adjust signal spacing to be of equal width across all times. There are two important parameters in this process:\n",
        "- interval: an interger that refers to the time span to compute aggregation of.\n",
        "- method: what aggregation method should be used to compute the value, by default this set to the mean.\n",
        "\n",
        "In addition to, passing the array of values and which column holds the values we wish to alter the frequency of."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hm0j0VPMuGuu"
      },
      "source": [
        "def time_segments_aggregate(X, interval, time_column, method=['mean']):\n",
        "    \"\"\"Aggregate values over given time span.\n",
        "    Args:\n",
        "        X (ndarray or pandas.DataFrame):\n",
        "            N-dimensional sequence of values.\n",
        "        interval (int):\n",
        "            Integer denoting time span to compute aggregation of.\n",
        "        time_column (int):\n",
        "            Column of X that contains time values.\n",
        "        method (str or list):\n",
        "            Optional. String describing aggregation method or list of strings describing multiple\n",
        "            aggregation methods. If not given, `mean` is used.\n",
        "    Returns:\n",
        "        ndarray, ndarray:\n",
        "            * Sequence of aggregated values, one column for each aggregation method.\n",
        "            * Sequence of index values (first index of each aggregated segment).\n",
        "    \"\"\"\n",
        "    if isinstance(X, np.ndarray):\n",
        "        X = pd.DataFrame(X)\n",
        "\n",
        "    X = X.sort_values(time_column).set_index(time_column)\n",
        "\n",
        "    if isinstance(method, str):\n",
        "        method = [method]\n",
        "\n",
        "    start_ts = X.index.values[0]\n",
        "    max_ts = X.index.values[-1]\n",
        "\n",
        "    values = list()\n",
        "    index = list()\n",
        "    while start_ts <= max_ts:\n",
        "        end_ts = start_ts + interval\n",
        "        subset = X.loc[start_ts:end_ts - 1]\n",
        "        aggregated = [\n",
        "            getattr(subset, agg)(skipna=True).values\n",
        "            for agg in method\n",
        "        ]\n",
        "        values.append(np.concatenate(aggregated))\n",
        "        index.append(start_ts)\n",
        "        start_ts = end_ts\n",
        "\n",
        "    return np.asarray(values), np.asarray(index)\n",
        "\n",
        "X, index = time_segments_aggregate(df, interval=1800, time_column='timestamp')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FVVq2U-cuGuu"
      },
      "source": [
        "If we go back to the source of the NYC Taxi data, we find that it records a value each 30 minutes. In the timestamp world, this is equivalent to 1800 seconds, therefore we set the interval to be `1800`. We also opt for the default aggregation method which is taking the `mean` value of each interval.\n",
        "\n",
        "Technically speaking, in our example the data is perfectly spaced, so we can skip this preprocessing step. However, that is not always the case and so we include it as a preprocessing primitive in the general pipeline as you will see later on."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "23tJs1SNuGuu"
      },
      "source": [
        "#### B. data imputation\n",
        "\n",
        "impute missing values that appear within the signal using scikit-learn's `SimpleImputer` which fills missing values by the mean value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "atjCiXZFuGuv"
      },
      "source": [
        "imp = SimpleImputer()\n",
        "X = imp.fit_transform(X)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ozfCAd5-uGuv"
      },
      "source": [
        "#### C. data normalization \n",
        "normalize the data between a specific range, we use scikit-learn's `MinMaxScaler` to scale data between [-1, 1]."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yfPtzr2RuGuv"
      },
      "source": [
        "scaler = MinMaxScaler(feature_range=(-1, 1))\n",
        "X = scaler.fit_transform(X)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eyqwS_9fuGuv"
      },
      "source": [
        "Notice how the y-axis changed after normalizing the data between [-1, 1]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXqvG1tsuGuv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 259
        },
        "outputId": "24dd44f3-abd0-4d60-baed-17dcd3f929f9"
      },
      "source": [
        "plot_ts(X)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "[encoded data, 286384 characters]\n",
            "text/plain": [
              "<Figure size 2160x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2MuVKxv_uGuw"
      },
      "source": [
        "#### D. slice the data to rolling window\n",
        "to prepare the data, we need to transform it into a sequence that is ingestable by the machine learning model. \n",
        "We take the signal we're interested in analyzing and we generate training examples. These training examples are mere snapshots of signal at different times.\n",
        "\n",
        "In order to do that, we adopt the sliding window approach of choosing a window of a pre-specified width and a particular step size. Once that's been decided we divide the signal indo segments, similar to what is depicited in the illustration below.\n",
        "\n",
        "<img src=\"https://github.com/signals-dev/Orion/blob/master/notebooks/tulog/images/sliding_window.png?raw=1\" alt=\"Drawing\" width=\"400\"/>\n",
        "\n",
        "We create a `rolling_window_sequence` function that slices the data into parts, each part contains:\n",
        "- the target value; the value at time t.\n",
        "- previous observed values, this is determined by the window width."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Y8PEaayuGuw"
      },
      "source": [
        "def rolling_window_sequences(X, index, window_size, target_size, step_size, target_column,\n",
        "                             drop=None, drop_windows=False):\n",
        "    \"\"\"Create rolling window sequences out of time series data.\n",
        "    The function creates an array of input sequences and an array of target sequences by rolling\n",
        "    over the input sequence with a specified window.\n",
        "    Optionally, certain values can be dropped from the sequences.\n",
        "    Args:\n",
        "        X (ndarray):\n",
        "            N-dimensional sequence to iterate over.\n",
        "        index (ndarray):\n",
        "            Array containing the index values of X.\n",
        "        window_size (int):\n",
        "            Length of the input sequences.\n",
        "        target_size (int):\n",
        "            Length of the target sequences.\n",
        "        step_size (int):\n",
        "            Indicating the number of steps to move the window forward each round.\n",
        "        target_column (int):\n",
        "            Indicating which column of X is the target.\n",
        "        drop (ndarray or None or str or float or bool):\n",
        "            Optional. Array of boolean values indicating which values of X are invalid, or value\n",
        "            indicating which value should be dropped. If not given, `None` is used.\n",
        "        drop_windows (bool):\n",
        "            Optional. Indicates whether the dropping functionality should be enabled. If not\n",
        "            given, `False` is used.\n",
        "    Returns:\n",
        "        ndarray, ndarray, ndarray, ndarray:\n",
        "            * input sequences.\n",
        "            * target sequences.\n",
        "            * first index value of each input sequence.\n",
        "            * first index value of each target sequence.\n",
        "    \"\"\"\n",
        "    out_X = list()\n",
        "    out_y = list()\n",
        "    X_index = list()\n",
        "    y_index = list()\n",
        "    target = X[:, target_column]\n",
        "\n",
        "    if drop_windows:\n",
        "        if hasattr(drop, '__len__') and (not isinstance(drop, str)):\n",
        "            if len(drop) != len(X):\n",
        "                raise Exception('Arrays `drop` and `X` must be of the same length.')\n",
        "        else:\n",
        "            if isinstance(drop, float) and np.isnan(drop):\n",
        "                drop = np.isnan(X)\n",
        "            else:\n",
        "                drop = X == drop\n",
        "\n",
        "    start = 0\n",
        "    max_start = len(X) - window_size - target_size + 1\n",
        "    while start < max_start:\n",
        "        end = start + window_size\n",
        "\n",
        "        if drop_windows:\n",
        "            drop_window = drop[start:end + target_size]\n",
        "            to_drop = np.where(drop_window)[0]\n",
        "            if to_drop.size:\n",
        "                start += to_drop[-1] + 1\n",
        "                continue\n",
        "\n",
        "        out_X.append(X[start:end])\n",
        "        out_y.append(target[end:end + target_size])\n",
        "        X_index.append(index[start])\n",
        "        y_index.append(index[end])\n",
        "        start = start + step_size\n",
        "\n",
        "    return np.asarray(out_X), np.asarray(out_y), np.asarray(X_index), np.asarray(y_index)\n",
        "\n",
        "X, y, X_index, y_index = rolling_window_sequences(X, index, \n",
        "                                                  window_size=100, \n",
        "                                                  target_size=1, \n",
        "                                                  step_size=1,\n",
        "                                                  target_column=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LFAxpKu8uGuw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c12c4e3d-d4ec-46ea-9507-b5be5a17c6b8"
      },
      "source": [
        "print(\"Training data input shape: {}\".format(X.shape))\n",
        "print(\"Training data index shape: {}\".format(X_index.shape))\n",
        "print(\"Training y shape: {}\".format(y.shape))\n",
        "print(\"Training y index shape: {}\".format(y_index.shape))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training data input shape: (10222, 100, 1)\n",
            "Training data index shape: (10222,)\n",
            "Training y shape: (10222, 1)\n",
            "Training y index shape: (10222,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-YSdE1XIuGux",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 165
        },
        "outputId": "16937f76-b1d8-4559-8467-eec582b7b104"
      },
      "source": [
        "plot_rws(X)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "[encoded data, 44324 characters]\n",
            "text/plain": [
              "<Figure size 1080x288 with 5 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e-snKuxHuGux"
      },
      "source": [
        "Where `X` represents the input used to train the model. In the previous example, we see `X` has `10222` training data points. Notice that `100` represents the window size. On the other hand, `y` is the real signal after processing, which we will use later on to calculate the error between the reconstructed and real signal. \n",
        "\n",
        "\n",
        "### Pipeline Training and Detection\n",
        "\n",
        "The architecture of the model requires four neural networks:\n",
        "- `encoder`: maps `X` to its latent representation `Z`.\n",
        "- `generator`: maps the latent variable `Z` back to `X`, which we will denote later on as `X_hat`.\n",
        "- `criticX`: discriminates between `X` and `generator(Z)` or `X_hat`.\n",
        "- `criticZ`: discriminates between `Z` and `encoder(X)`.\n",
        "\n",
        "we detail the composition of each network in `model.py`.\n",
        "\n",
        "To use the `TadGAN` model, we specify a number of parameters including the model layers (structure of the previously mentioned neural networks). We also specify the input dimensions, the number of epochs, the learning rate, etc. All the parameters are listed below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bDfG62KsuGux",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 701
        },
        "outputId": "e964ae0e-82a1-4fc0-b202-a445baa66d0b"
      },
      "source": [
        "from model import hyperparameters\n",
        "from orion.primitives.tadgan import TadGAN\n",
        "\n",
        "hyperparameters[\"epochs\"] = 100\n",
        "hyperparameters[\"shape\"] = (100, 1) # based on the window size\n",
        "hyperparameters[\"optimizer\"] = \"keras.optimizers.Adam\"\n",
        "hyperparameters[\"learning_rate\"] = 0.0005\n",
        "hyperparameters[\"latent_dim\"] = 20\n",
        "hyperparameters[\"batch_size\"] = 64\n",
        "\n",
        "tgan = TadGAN(**hyperparameters)\n",
        "tgan.fit(X)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py:297: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set `model.trainable` without calling `model.compile` after ?\n",
            "  'Discrepancy between trainable weights and collected trainable'\n",
            "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py:297: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set `model.trainable` without calling `model.compile` after ?\n",
            "  'Discrepancy between trainable weights and collected trainable'\n",
            "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py:297: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set `model.trainable` without calling `model.compile` after ?\n",
            "  'Discrepancy between trainable weights and collected trainable'\n",
            "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py:297: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set `model.trainable` without calling `model.compile` after ?\n",
            "  'Discrepancy between trainable weights and collected trainable'\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 1/100, [Dx loss: [-0.79510003 -6.622919    5.137222    0.06905992]] [Dz loss: [-1.248113   -1.2148395  -1.3372226   0.13039489]] [G loss: [-1.7341048  -5.1221642   2.0607867   0.13272728]]\n",
            "Epoch: 2/100, [Dx loss: [-0.5293686  -7.0624237   6.4027658   0.01302929]] [Dz loss: [-2.4776664  -1.6896737  -1.6903356   0.09023429]] [G loss: [-2.936392   -6.4008327   2.263435    0.12010046]]\n",
            "Epoch: 3/100, [Dx loss: [ -0.5295739  -10.5860615    9.925359     0.01311276]] [Dz loss: [-1.4918855  -1.1455413  -0.7994108   0.04530665]] [G loss: [-7.8890343  -9.928858    1.2801272   0.07596967]]\n",
            "Epoch: 4/100, [Dx loss: [ -0.32507813 -10.11016      9.649068     0.0136011 ]] [Dz loss: [-0.70832956  0.01039923 -1.1832917   0.04645638]] [G loss: [-7.0611095 -9.742879   1.8956887  0.0786079]]\n",
            "Epoch: 5/100, [Dx loss: [-5.09258687e-01 -1.27581415e+01  1.21465034e+01  1.02379881e-02]] [Dz loss: [-3.862498    2.880299   -8.578554    0.18357563]] [G loss: [ -0.23098886 -11.903468    10.27893      0.13935493]]\n",
            "Epoch: 6/100, [Dx loss: [-5.0288814e-01 -1.1951165e+01  1.1347123e+01  1.0115872e-02]] [Dz loss: [-3.7369554  3.22506   -8.458705   0.1496692]] [G loss: [ -0.61651117 -11.332743     9.740089     0.09761447]]\n",
            "Epoch: 7/100, [Dx loss: [-0.43709338 -9.184546    8.645328    0.01021279]] [Dz loss: [ 1.2048544   2.1377702  -1.3314245   0.03985097]] [G loss: [-6.5612783  -8.702479    1.4544153   0.06867866]]\n",
            "Epoch: 8/100, [Dx loss: [-3.2715479e-01 -1.5956944e+01  1.5536484e+01  9.3307812e-03]] [Dz loss: [ 1.0677394   1.4156266  -0.9114192   0.05635316]] [G loss: [-13.359826   -15.388186     1.1503425    0.08780169]]\n",
            "Epoch: 9/100, [Dx loss: [-2.8917518e-01 -1.5606696e+01  1.5240568e+01  7.6946914e-03]] [Dz loss: [ 0.18491572  0.43412533 -0.92806387  0.06788545]] [G loss: [-13.153973   -15.263011     1.355741     0.07532998]]\n",
            "Epoch: 10/100, [Dx loss: [-2.4090266e-01 -8.3478251e+00  8.0419750e+00  6.4946446e-03]] [Dz loss: [ 0.21527518  0.7714591  -1.1474036   0.05912194]] [G loss: [-5.712921   -7.909572    1.5778997   0.06187497]]\n",
            "Epoch: 11/100, [Dx loss: [-3.9095196e-01 -1.7246012e+01  1.6785378e+01  6.9683958e-03]] [Dz loss: [-0.32157043  0.43772322 -1.2553846   0.04960909]] [G loss: [-14.86662   -16.88036     1.5325011   0.0481239]]\n",
            "Epoch: 12/100, [Dx loss: [-2.9591599e-01 -1.3978735e+01  1.3612998e+01  6.9819842e-03]] [Dz loss: [ 0.5915169   1.0382168  -1.0568265   0.06101268]] [G loss: [-11.638993   -13.404485     1.2971851    0.04683068]]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-17-f061bf534063>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0mtgan\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTadGAN\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mhyperparameters\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m \u001b[0mtgan\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/orion/primitives/tadgan.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, **kwargs)\u001b[0m\n\u001b[1;32m    250\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_build_tadgan\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    251\u001b[0m         \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 252\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    253\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    254\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/orion/primitives/tadgan.py\u001b[0m in \u001b[0;36m_fit\u001b[0;34m(self, X)\u001b[0m\n\u001b[1;32m    233\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    234\u001b[0m                 epoch_g_loss.append(\n\u001b[0;32m--> 235\u001b[0;31m                     self.encoder_generator_model.train_on_batch([x, z], [valid, valid, x]))\n\u001b[0m\u001b[1;32m    236\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    237\u001b[0m             \u001b[0mcx_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepoch_cx_loss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mtrain_on_batch\u001b[0;34m(self, x, y, sample_weight, class_weight, reset_metrics)\u001b[0m\n\u001b[1;32m   1512\u001b[0m             \u001b[0mins\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0msample_weights\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1513\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_train_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1514\u001b[0;31m         \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1515\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1516\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mreset_metrics\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/backend.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   3474\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3475\u001b[0m     fetched = self._callable_fn(*array_vals,\n\u001b[0;32m-> 3476\u001b[0;31m                                 run_metadata=self.run_metadata)\n\u001b[0m\u001b[1;32m   3477\u001b[0m     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_fetch_callbacks\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfetched\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3478\u001b[0m     output_structure = nest.pack_sequence_as(\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1470\u001b[0m         ret = tf_session.TF_SessionRunCallable(self._session._session,\n\u001b[1;32m   1471\u001b[0m                                                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_handle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1472\u001b[0;31m                                                run_metadata_ptr)\n\u001b[0m\u001b[1;32m   1473\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1474\u001b[0m           \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T88nkZzOuGux"
      },
      "source": [
        "# reconstruct\n",
        "X_hat, critic = tgan.predict(X)\n",
        "\n",
        "# visualize X_hat\n",
        "plot_rws(X_hat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pQSc7gvQuGuy"
      },
      "source": [
        "To reassemble or “unroll” the predicted signal `X_hat` we can choose different aggregation methods (e.g., mean, max, etc). In our implementation, we chose it to as the median value.\n",
        "\n",
        "<img src=\"https://github.com/signals-dev/Orion/blob/master/notebooks/tulog/images/reconstruct.png?raw=1\" alt=\"Drawing\" style=\"width: 1000px;\"/>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8426wtaHuGuy"
      },
      "source": [
        "# flatten the predicted windows\n",
        "y_hat = unroll_ts(X_hat)\n",
        "\n",
        "# plot the time series\n",
        "plot_ts([y, y_hat], labels=['original', 'reconstructed'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CBnN99o1uGuy"
      },
      "source": [
        "We can see that the GAN model did really well in trying to reconstruct the signal. We also see how it expected the signal to be, in comparison to what it actually is. The discrepancies between the two signals will be used to calculate the error. The higher the error, the more likely it is an anomaly"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5PAx3CzyuGuy"
      },
      "source": [
        "# pair-wise error calculation\n",
        "error = np.zeros(shape=y.shape)\n",
        "length = y.shape[0]\n",
        "for i in range(length):\n",
        "    error[i] = abs(y_hat[i] - y[i])\n",
        "\n",
        "# visualize the error curve\n",
        "fig = plt.figure(figsize=(30, 3))\n",
        "plt.plot(error)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7N3-oyLcuGuy"
      },
      "source": [
        "### Error Computation\n",
        "\n",
        "In the `TadGAN` pipeline, we use `tadgan.score_anomalies` to perform error calculation for us. It is a smoothed error function that uses a window based method to smooth the curve then uses either: area, point difference, or dtw as a measure of discrepancy. \n",
        "\n",
        "#### Area\n",
        "This method captures the general shape of the orignal and reconstructed signal and then compares them together. \n",
        "\n",
        "<img src=\"https://github.com/signals-dev/Orion/blob/master/notebooks/tulog/images/area.png?raw=1\" alt=\"Drawing\" width=\"300\"/>\n",
        "\n",
        "#### Point\n",
        "This method applies a point-to-point comparison between the original and reconstructed signal. It is considered a strict approach that does not allow for many mistakes.\n",
        "\n",
        "<img src=\"https://github.com/signals-dev/Orion/blob/master/notebooks/tulog/images/point.png?raw=1\" alt=\"Drawing\" width=\"180\"/>\n",
        "\n",
        "#### DTW\n",
        "A more lenient method yet very effective is Dynamic Time Warping (DTW). It compares two signals together using any pair-wise distance measure but it allows for one signal to be lagging behind another.\n",
        "\n",
        "<img src=\"https://github.com/signals-dev/Orion/blob/master/notebooks/tulog/images/dtw.png?raw=1\" alt=\"Drawing\" width=\"400\"/>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6G7iD0NguGuy"
      },
      "source": [
        "from orion.primitives.tadgan import score_anomalies\n",
        "\n",
        "error, true_index, true, pred = score_anomalies(X, X_hat, critic, X_index, rec_error_type=\"dtw\", comb=\"mult\")\n",
        "pred = np.array(pred).mean(axis=2)\n",
        "\n",
        "# visualize the error curve\n",
        "plot_error([[true, pred], error])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bHJMzAZ8uGuz"
      },
      "source": [
        "Now we can visually see where the error reaches a substantially high value. But how should we decide if the error value determines a potential anomaly? We could use a fixed threshold that says if `error > 10` then let’s classify the datapoint as anomalous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6mDss13-uGuz"
      },
      "source": [
        "# threshold\n",
        "thresh = 10\n",
        "\n",
        "intervals = list()\n",
        "\n",
        "i = 0\n",
        "max_start = len(error)\n",
        "while i < max_start:\n",
        "    j = i\n",
        "    start = index[i]\n",
        "    while error[i] > thresh:\n",
        "        i += 1\n",
        "    \n",
        "    end = index[i]\n",
        "    if start != end:\n",
        "        intervals.append((start, end, np.mean(error[j: i+1])))\n",
        "        \n",
        "    i += 1\n",
        "        \n",
        "intervals"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zB8RNnNguGuz"
      },
      "source": [
        "anomalies = pd.DataFrame(intervals, columns=['start', 'end', 'score'])\n",
        "plot(df, [anomalies, known_anomalies])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H6MDyhBWuGu0"
      },
      "source": [
        "While a fixed threshold raised two correct anomalies, it missed out on the other three. If we were to look back at the error plot, we notice that some deviations are abnormal within its local region. So how can we incorporate this information in our thresholding technique? We can use window based methods to detect anomalies with respect to their context.\n",
        "\n",
        "We first define the window of errors, that we want to analyze. We then find the anomalous\n",
        "sequences in that window by looking at the mean and standard deviation of the errors in the window.\n",
        "We store the start/stop index pairs that correspond to each sequence, along with its score. \n",
        "We then move the window and repeat the procedure.\n",
        "Lastly, we combine overlapping or consecutive sequences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kudu1tfbuGu0"
      },
      "source": [
        "from orion.primitives.timeseries_anomalies import find_anomalies\n",
        "\n",
        "# find anomalies\n",
        "intervals = find_anomalies(error, index, \n",
        "                           window_size_portion=0.33, \n",
        "                           window_step_size_portion=0.1, \n",
        "                           fixed_threshold=True)\n",
        "intervals"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AAM8aw0KuGu0"
      },
      "source": [
        "# visualize the result\n",
        "anomalies = pd.DataFrame(intervals, columns=['start', 'end', 'score'])\n",
        "plot(df, [anomalies, known_anomalies])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ShSMiPP_uGu0"
      },
      "source": [
        "Cool! We now have the same result as we saw previously. The red intervals depict the detected anomalies, the green intervals show the ground truth. Several anomalies were detected. We also see that it detected some other intervals that were not included in the ground truth labels.\n",
        "\n",
        "Using the Orion API and pipelines, we simplified this process yet allowed flexibility for pipeline configuration.\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vj-V2GwIuGu0"
      },
      "source": [
        "### End-to-End Pipeline Configuration\n",
        "\n",
        "To configure a pipeline, we adjust the parameters of the primitive of interest within the `pipeline.json` file or directly by passing the dictionary to the API.\n",
        "\n",
        "In the following example, I changed the aggregation level as well as the number of `epochs` for training. These changes will override the parameters specified in the `.json` file. To know more about the API usage and primitive designs, please refer to the [documentation](https://signals-dev.github.io/Orion)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-IBMGoXWuGu1"
      },
      "source": [
        "from orion import Orion\n",
        "\n",
        "parameters = {\n",
        "    \"mlprimitives.custom.timeseries_preprocessing.time_segments_aggregate#1\": {\n",
        "            \"interval\": 3600 # hour level\n",
        "        },\n",
        "    'orion.primitives.tadgan.TadGAN#1': {\n",
        "        'epochs': 15,\n",
        "        }\n",
        "}\n",
        "\n",
        "orion = Orion(\n",
        "    'tadgan.json',\n",
        "    parameters\n",
        ")\n",
        "\n",
        "anomalies = orion.fit_detect(df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8e1913V_uGu1"
      },
      "source": [
        "plot(df, [anomalies, known_anomalies])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7vPbbpKYuGu1"
      },
      "source": [
        "The anomalies detected in this run are a bit different from the earlier example. Although, it still succeeded in detecting anomalies. Maybe a 1 hour aggregate is not the appropriate value? Maybe we did not train the model enough times, or maybe too many times... How can we tell? One way is to look at the result of the model like we have done previously.\n",
        "\n",
        "You can use the `visualization` parameter within `detect` to return intermediate outputs (primitive outputs) that we are interested in. For example in the `tadgan.json` file, use `visualization` to return the following variables:\n",
        "\n",
        "- `X`: this is the output of the preprocessing steps from averaging, imputing, and scaling. These steps were showcased previously as steps (A, B, and C).\n",
        "- `X_hat`: this is the \"predicted\" output by the __TadGAN__ model without any processing. It represents the reconstructed window at each timepoint.\n",
        "- `es`: this is the error calculated by capturing the discrepancies between original and reconstructed signal.\n",
        "\n",
        "then we use `anomalies, viz = orion.detect(df, visualization=True)` where `viz` will be a dictionary of intermediate outputs.\n",
        "\n",
        "**Note**: we will talk more about how to evaluate the detected anommalies with respect to the ground truth in part 3 of the tutorial.\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k8TBuTe-uGu1"
      },
      "source": [
        "## Part 3\n",
        "\n",
        "In part three of the series, we look at evaluation and end-to-end anomaly detection evaluation. \n",
        "\n",
        "### Evaluation\n",
        "\n",
        "We compare the anomalies given to us as ground truth labels to the detected anomalies. But first, we look at some of the mechanisms we have for evaluation, namely:\n",
        "* **weighted segment**: Assessing every datapoint in the detected anomalies with its counterpart in the ground truth.\n",
        "* **overlap segment**: Assesses the detected anomaly segment by seeing if we caught an overlap with the correct anomalies.\n",
        "\n",
        "We will look at both approaches, but first let's construct a dummy dataset.\n",
        "\n",
        "<img src=\"https://github.com/signals-dev/Orion/blob/master/notebooks/tulog/images/truth_predicted.jpg?raw=1\" alt=\"Drawing\" style=\"width: 600px;\"/>\n",
        "\n",
        "Let's assume that the signal starts at timestamp 1, and ends at timestamp 20. We can then see that the ground truth contains three anomalies, namely `(5, 8)`, `(12, 13)`, and `(17, 18)`, where `(i, j)` expresses the starting timestamp `i` and ending timestamp `j`.\n",
        "\n",
        "We can also see that, we detected two anomalies, namely `(5, 8)` and `(12, 15)`. So how can we compare both sets?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hFbSRfhJuGu1"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "# to reproduce the same dummy signal\n",
        "np.random.seed(0)\n",
        "\n",
        "# dummy data\n",
        "start, end = (1, 20)\n",
        "signal = np.random.rand(end - start, 1)\n",
        "\n",
        "ground_truth = [\n",
        "    (5, 8),\n",
        "    (12, 13),\n",
        "    (17, 18)\n",
        "]\n",
        "\n",
        "anomalies = [\n",
        "    (5, 8),\n",
        "    (12, 15)\n",
        "]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "svTr6PFruGu1"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "time = range(start, end)\n",
        "plt.plot(time, signal)\n",
        "\n",
        "# ground truth\n",
        "for i, (t1, t2) in enumerate(ground_truth):\n",
        "    plt.axvspan(t1, t2+1, color=\"g\", alpha=0.2, label=\"ground_truth\")\n",
        "\n",
        "# detected\n",
        "for i, (t1, t2) in enumerate(anomalies):\n",
        "    plt.axvspan(t1, t2+1, color=\"r\", alpha=0.2, label=\"detected\")\n",
        "\n",
        "    \n",
        "plt.title(\"Example\")\n",
        "plt.xlabel(\"Time\")\n",
        "plt.ylabel(\"value\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ysSzQh_6uGu2"
      },
      "source": [
        "There are two approaches for comparing anomaly sets, as expressed earlier.\n",
        "\n",
        "(1) **Weighted Segment**, a stricter method, it is valuable to use when you want to equalize the importance of detecting anomalies, and normal instances. \n",
        "* It first segments the signal into partitions based on the ground truth and detected sequences. \n",
        "* Then it makes a segment to segment comparison and records TP/FP/FN/TN accordingly. \n",
        "* The overall score is weighted by the duration (size) of each segment. \n",
        "\n",
        "Visually, this operation is summarized by the illustration below. \n",
        "\n",
        "<img src=\"https://github.com/signals-dev/Orion/blob/master/notebooks/tulog/images/weighted_segment.jpg?raw=1\" alt=\"Drawing\" style=\"width: 700px;\"/>\n",
        "\n",
        "we can use `orion.evaluation` subpackage to compute multiple metrics using the weighted segment approach. For example to compute the accuracy, we use `contextual_accuracy(..., weighted=True)`. There are other metrics available, for reference checkout the `orion.evaluation` documentation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZZ2E45h4uGu2"
      },
      "source": [
        "from orion.evaluation.contextual import contextual_accuracy, contextual_f1_score\n",
        "\n",
        "accuracy = contextual_accuracy(ground_truth, anomalies, start=start, end=end)\n",
        "f1_score = contextual_f1_score(ground_truth, anomalies, start=start, end=end)\n",
        "\n",
        "print(\"Accuracy score = {:0.3f}\".format(accuracy))\n",
        "print(\"F1 score = {:0.3f}\".format(f1_score))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VtzgZ8fpuGu4"
      },
      "source": [
        "(2) **Overlap Segment**, a more lenient approach of evaluation. It takes the perspective of rewarding the system if it manages to alarm the user of a subset of an anomaly. More particularly, it records:\n",
        "\n",
        "1) TP, if a ground truth segment overlaps with the detected segment. \n",
        "\n",
        "2) FN, If the ground truth segment does not overlap any detected segments. \n",
        "\n",
        "3) FP, If a detected  segment does not overlap any labeled anomalous region. \n",
        "\n",
        "This can be summarized by the illustration below.\n",
        "\n",
        "<img src=\"https://github.com/signals-dev/Orion/blob/master/notebooks/tulog/images/overlap_segment.jpg?raw=1\" alt=\"Drawing\" style=\"width: 700px;\"/>\n",
        "\n",
        "Similarly, we can use the same metric functions, but this time we use the parameter `weighted=False`. Note: overlap segment approach, does not account for true negatives. Reason being, anomalies in time series data are rare and so \"normal\" instances will skew the value of the computed metric. Therefore, using this approach we cannot compute metrics such as the `accuracy`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bltjh3OzuGu4"
      },
      "source": [
        "f1_score = contextual_f1_score(ground_truth, anomalies, start=start, end=end, weighted=False)\n",
        "\n",
        "print(\"F1 score = {:0.3f}\".format(f1_score))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "89IIpQU4uGu4"
      },
      "source": [
        "### Pipeline evaluation end-to-end \n",
        "\n",
        "We integrate the evaluation suite into the Orion API, such that you can evaluate the pipeline on a dataset (with its labels) end-to-end. \n",
        "\n",
        "Following part 2 introduction to the Orion API, we can create an `orion` instance the use its `evaluate` functionality. We support the method with the following arguments:\n",
        "* `data`, a `pandas.DataFrame` containing two columns: timestamp and value.\n",
        "* `truth`, a `pandas.DataFrame` containing two columns: start timestamp and end timestamp of ground truth labels \n",
        "* `fit`, a flag denoting whether to train the pipeline before evaluating it.\n",
        "train_data, a `pandas.DataFrame` containing two columns: timestamp and value used to train the pipeline, if not given, the pipeline will be trained on `data`.\n",
        "* `metrics`, a list of metrics used to evaluate the pipeline. \n",
        "\n",
        "In the previous part we went through how to train a pipeline and use it for anomaly detection, the focus now is on defining `metrics` and evaluating the performance of the pipeline.\n",
        "\n",
        "`metrics` is list of function names that compares a ground truth labels against detected labels and returns a metric value. We have seen some functions of that sort, such as `contextual_accuracy` and `contextual_f1_score`. To construct our metrics list we select some of the predefined metrics in Orion, such as:\n",
        "* Accuracy\n",
        "* F1 Score\n",
        "* Precision\n",
        "* Recall\n",
        "\n",
        "By default, we use the `weighted segment` approach, you can override metrics defined by specifying aa new `metrics` dictitonary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OzM0a8kuuGu5"
      },
      "source": [
        "from orion import Orion\n",
        "from orion.data import load_signal, load_anomalies\n",
        "from orion.evaluation.contextual import contextual_accuracy, contextual_f1_score, contextual_precision\n",
        "\n",
        "metrics = [\n",
        "    'f1',\n",
        "    'recall',\n",
        "    'precision',\n",
        "]\n",
        "\n",
        "orion = Orion(\n",
        "    'tadgan.json'\n",
        ")\n",
        "\n",
        "signal = 'nyc_taxi'\n",
        "\n",
        "# load signal\n",
        "df = load_signal(signal)\n",
        "\n",
        "# load ground truth anomalies\n",
        "ground_truth = load_anomalies(signal)\n",
        "\n",
        "scores = orion.evaluate(df, ground_truth, fit=True, metrics=metrics)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ux6HSXw-uGu5"
      },
      "source": [
        "scores"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tJaezrtE1uDR"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    }
  ]
}