{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "3D-Ken-Burns.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrm8488/shared_colab_notebooks/blob/master/3D_Ken_Burns.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uCEixJujApc8",
        "colab_type": "text"
      },
      "source": [
        "# 3D Ken Burns\n",
        "This is a reference implementation of 3D Ken Burns Effect from a Single Image [1] using PyTorch. Given a single input image, it animates this still image with a virtual camera scan and zoom subject to motion parallax. Should you be making use of our work, please cite our paper [1].\n",
        "\n",
        "<a href=\"https://arxiv.org/abs/1909.05483\" rel=\"Paper\"><img src=\"http://content.sniklaus.com/kenburns/paper.jpg\" alt=\"Paper\" width=\"100%\"></a>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cJFpObViMXDH",
        "colab_type": "text"
      },
      "source": [
        "> Colab created by [Manuel Romero](https://twitter.com/mrm8488)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YXK9VNecNQEF",
        "colab_type": "text"
      },
      "source": [
        "### 1. Clone the project"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZbYRw_GlCL5w",
        "colab_type": "code",
        "outputId": "d8663687-1123-4ca4-f455-a75af5dfb5c3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        }
      },
      "source": [
        "!git clone https://github.com/sniklaus/3d-ken-burns.git"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into '3d-ken-burns'...\n",
            "remote: Enumerating objects: 26, done.\u001b[K\n",
            "remote: Counting objects: 100% (26/26), done.\u001b[K\n",
            "remote: Compressing objects: 100% (22/22), done.\u001b[K\n",
            "remote: Total 26 (delta 6), reused 22 (delta 4), pack-reused 0\u001b[K\n",
            "Unpacking objects: 100% (26/26), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "06tpWil4A6Ck",
        "colab_type": "text"
      },
      "source": [
        "### 2. Download pretrained models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jy-x7Cm7AymM",
        "colab_type": "code",
        "outputId": "da328035-e772-4282-a55c-cf1855461df2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 604
        }
      },
      "source": [
        "!wget -O ./3d-ken-burns/models/disparity-estimation.pytorch http://content.sniklaus.com/kenburns/network-disparity.pytorch\n",
        "!wget -O ./3d-ken-burns/models/disparity-refinement.pytorch http://content.sniklaus.com/kenburns/network-refinement.pytorch\n",
        "!wget -O ./3d-ken-burns/models/pointcloud-inpainting.pytorch http://content.sniklaus.com/kenburns/network-inpainting.pytorch"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-11-22 18:13:23--  http://content.sniklaus.com/kenburns/network-disparity.pytorch\n",
            "Resolving content.sniklaus.com (content.sniklaus.com)... 51.15.58.24\n",
            "Connecting to content.sniklaus.com (content.sniklaus.com)|51.15.58.24|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 357438463 (341M) [application/octet-stream]\n",
            "Saving to: ‘./3d-ken-burns/models/disparity-estimation.pytorch’\n",
            "\n",
            "./3d-ken-burns/mode 100%[===================>] 340.88M  9.65MB/s    in 35s     \n",
            "\n",
            "2019-11-22 18:13:58 (9.86 MB/s) - ‘./3d-ken-burns/models/disparity-estimation.pytorch’ saved [357438463/357438463]\n",
            "\n",
            "--2019-11-22 18:14:00--  http://content.sniklaus.com/kenburns/network-refinement.pytorch\n",
            "Resolving content.sniklaus.com (content.sniklaus.com)... 51.15.58.24\n",
            "Connecting to content.sniklaus.com (content.sniklaus.com)|51.15.58.24|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2438928 (2.3M) [application/octet-stream]\n",
            "Saving to: ‘./3d-ken-burns/models/disparity-refinement.pytorch’\n",
            "\n",
            "./3d-ken-burns/mode 100%[===================>]   2.33M  1.26MB/s    in 1.8s    \n",
            "\n",
            "2019-11-22 18:14:02 (1.26 MB/s) - ‘./3d-ken-burns/models/disparity-refinement.pytorch’ saved [2438928/2438928]\n",
            "\n",
            "--2019-11-22 18:14:04--  http://content.sniklaus.com/kenburns/network-inpainting.pytorch\n",
            "Resolving content.sniklaus.com (content.sniklaus.com)... 51.15.58.24\n",
            "Connecting to content.sniklaus.com (content.sniklaus.com)|51.15.58.24|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 33179067 (32M) [application/octet-stream]\n",
            "Saving to: ‘./3d-ken-burns/models/pointcloud-inpainting.pytorch’\n",
            "\n",
            "./3d-ken-burns/mode 100%[===================>]  31.64M  6.96MB/s    in 4.5s    \n",
            "\n",
            "2019-11-22 18:14:09 (6.96 MB/s) - ‘./3d-ken-burns/models/pointcloud-inpainting.pytorch’ saved [33179067/33179067]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EkrEO7vIIPym",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "os.environ['CUDA_HOME'] = \"/usr/local/cuda\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vtd0th84B0Zg",
        "colab_type": "code",
        "outputId": "aa96e083-2bca-431a-aec0-6c39233c623f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 141
        }
      },
      "source": [
        "!pip install -q moviepy gevent"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: moviepy in /usr/local/lib/python3.6/dist-packages (0.2.3.5)\n",
            "Requirement already satisfied: decorator<5.0,>=4.0.2 in /usr/local/lib/python3.6/dist-packages (from moviepy) (4.4.1)\n",
            "Requirement already satisfied: tqdm<5.0,>=4.11.2 in /usr/local/lib/python3.6/dist-packages (from moviepy) (4.28.1)\n",
            "Requirement already satisfied: imageio<3.0,>=2.1.2 in /usr/local/lib/python3.6/dist-packages (from moviepy) (2.4.1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from moviepy) (1.17.4)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from imageio<3.0,>=2.1.2->moviepy) (4.3.0)\n",
            "Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from pillow->imageio<3.0,>=2.1.2->moviepy) (0.46)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xXFs3ii5Co82",
        "colab_type": "code",
        "outputId": "1cd39f5f-504b-4a53-9315-e15c852d8baf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "%cd /content/3d-ken-burns"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/3d-ken-burns\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ndl8FGKxM_hk",
        "colab_type": "text"
      },
      "source": [
        "### 3. Upload a picture from your filesystem"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wgRSDEfRJ6ha",
        "colab_type": "code",
        "outputId": "5d75e1ec-f86d-446a-8e1d-7f4de151469a",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "[encoded data, 7324 characters]",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 92
        }
      },
      "source": [
        "from google.colab import files\n",
        "\n",
        "uploaded = files.upload()\n",
        "pic_name = list(uploaded.keys())[0]\n",
        "print(pic_name)\n",
        "!mv ./\"$pic_name\" ./images/$pic_name"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-4d96a24b-c9eb-4820-850c-d7fc1977e7b2\" name=\"files[]\" multiple disabled />\n",
              "     <output id=\"result-4d96a24b-c9eb-4820-850c-d7fc1977e7b2\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving doublestrike.jpg to doublestrike.jpg\n",
            "doublestrike.jpg\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Daw9Ptv5P_EG",
        "colab_type": "text"
      },
      "source": [
        "### 4. Apply 3D Ken Burns Effect"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H_6mgeGBDdVD",
        "colab_type": "code",
        "outputId": "9fbad4d6-3817-4c26-9640-fd46dbdbf6fc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 304
        }
      },
      "source": [
        "!python autozoom.py --in ./images/\"$pic_name\" --out ./autozoom.mp4"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Imageio: 'ffmpeg-linux64-v3.3.1' was not found on your computer; downloading it now.\n",
            "Try 1. Download from https://github.com/imageio/imageio-binaries/raw/master/ffmpeg/ffmpeg-linux64-v3.3.1 (43.8 MB)\n",
            "Downloading: 8192/45929032 bytes (0.0%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b1228800/45929032 bytes (2.7%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b5095424/45929032 bytes (11.1%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b8577024/45929032 bytes (18.7%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b12427264/45929032 bytes (27.1%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b16105472/45929032 bytes (35.1%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b20103168/45929032 bytes (43.8%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b24051712/45929032 bytes (52.4%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b27721728/45929032 bytes (60.4%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b31694848/45929032 bytes (69.0%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b35545088/45929032 bytes (77.4%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b39460864/45929032 bytes (85.9%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b43163648/45929032 bytes (94.0%)\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b45929032/45929032 bytes (100.0%)\n",
            "  Done\n",
            "File saved as /root/.imageio/ffmpeg/ffmpeg-linux64-v3.3.1.\n",
            "Downloading: \"https://download.pytorch.org/models/vgg19_bn-c79401a0.pth\" to /root/.cache/torch/checkpoints/vgg19_bn-c79401a0.pth\n",
            "100% 548M/548M [00:19<00:00, 28.9MB/s]\n",
            "Downloading: \"https://download.pytorch.org/models/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth\" to /root/.cache/torch/checkpoints/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth\n",
            "100% 170M/170M [00:06<00:00, 25.6MB/s]\n",
            "[MoviePy] >>>> Building video ./autozoom.mp4\n",
            "[MoviePy] Writing video ./autozoom.mp4\n",
            "100% 150/150 [00:07<00:00, 20.86it/s]\n",
            "[MoviePy] Done.\n",
            "[MoviePy] >>>> Video ready: ./autozoom.mp4 \n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yxbbS6JxJZDl",
        "colab_type": "code",
        "outputId": "7de7c329-ff82-415e-c4f3-19c79f62835b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 470
        }
      },
      "source": [
        "from IPython.display import HTML\n",
        "from base64 import b64encode\n",
        "mp4 = open('./autozoom.mp4','rb').read()\n",
        "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
        "HTML(\"\"\"\n",
        "<video width=600 controls autoplay>\n",
        "      <source src=\"%s\" type=\"video/mp4\">\n",
        "</video>\n",
        "\"\"\" % data_url)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "<video width=600 controls autoplay>\n",
              "      <source src=\"data:video/mp4;base64,[encoded data, 684812 characters]\" type=\"video/mp4\">\n",
              "</video>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rudZ-R-YW67q",
        "colab_type": "text"
      },
      "source": [
        "### 4. Download it!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-ovDi2JrUz6o",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "files.download(\"./autozoom.mp4\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOyjsg55NeI0",
        "colab_type": "text"
      },
      "source": [
        "#### If you want to try with another picture go to step 3"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "72XSBhKkPKMA",
        "colab_type": "text"
      },
      "source": [
        " Did you enjoy? Buy me a coffe :)\n",
        " \n",
        "[![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/Y8Y3VYYE)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HuHR8hcF-tDW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [],
      "execution_count": 0,
      "outputs": []
    }
  ]
}
