{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "3D-Photo-Inpainting.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrm8488/shared_colab_notebooks/blob/master/3D_Photo_Inpainting.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IY-s3ZpHBeAU",
        "colab_type": "text"
      },
      "source": [
        "# 3D Photography using Context-aware Layered Depth Inpainting (CVPR 2020)\n",
        "\n",
        "[project website](https://shihmengli.github.io/3D-Photo-Inpainting/)\n",
        "\n",
        "> Colab adapted version by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LPQ0EWxgvMIc",
        "colab_type": "text"
      },
      "source": [
        "## 1. Install required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5o-EIMeaghU0",
        "colab_type": "code",
        "outputId": "c18d16cb-bea0-46d8-8eec-2cf924226d7b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "!pip3 install -q torch==1.4.0+cu100 torchvision==0.5.0+cu100 -f https://download.pytorch.org/whl/torch_stable.html\n",
        "!pip3 install -q opencv-python==4.2.0.32\n",
        "!pip3 install -q vispy==0.6.4\n",
        "!pip3 install -q moviepy==1.0.2\n",
        "!pip3 install -q transforms3d==0.3.1\n",
        "!pip3 install -q networkx==2.3\n",
        "!pip install -q PyDrive"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K     |████████████████████████████████| 2.3MB 4.6MB/s \n",
            "\u001b[K     |████████████████████████████████| 890kB 58.1MB/s \n",
            "\u001b[?25h"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c-g7AeLMvY0R",
        "colab_type": "text"
      },
      "source": [
        "## 2. Download script and pretrained model\n",
        "- It will ask for you a verification code\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n-Kl6fo1oows",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "![ -e 3d-photo-inpainting-master ] && rm -r 3d-photo-inpainting-master\n",
        "\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        "\n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)\n",
        "downloaded = drive.CreateFile({'id':\"1Jmiq1-hSIsHiq_GCcmiNXtrs_T62AY_z\"})\n",
        "downloaded.GetContentFile('3d-photo-inpainting-master.zip')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "seX7gKalBPiH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%bash\n",
        "unzip 3d-photo-inpainting-master.zip\n",
        "rm 3d-photo-inpainting-master.zip\n",
        "[ -e /content/3d-photo-inpainting-master/image/moon.jpg ] && rm -rf /content/3d-photo-inpainting-master/image/moon.jpg"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y3p-239gGKxy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c3c81ba6-be5e-438f-f5b4-294aed865643"
      },
      "source": [
        "cd 3d-photo-inpainting-master"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/3d-photo-inpainting-master\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d-pwf9Qzx5zI",
        "colab_type": "text"
      },
      "source": [
        "## 3. Upload a picture from your filesystem with jpg extension"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7tpMX25X8HYh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!rm -rf /content/3d-photo-inpainting-master/image/*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vmX2ZWus9UrU",
        "colab_type": "code",
        "outputId": "e6abb3ea-b492-4af4-ccf7-3f77d2eb716e",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "[encoded data, 7324 characters]",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 74
        }
      },
      "source": [
        "import os\n",
        "import shutil\n",
        "from google.colab import files\n",
        "\n",
        "uploaded = files.upload()\n",
        "pic_name = list(uploaded.keys())[0]\n",
        "\n",
        "if(pic_name.split(\".\")[1].lower() != \"jpg\"):\n",
        "  print(\"YOU MUST UPLOAD A .jpg FILE!! Re-run this cell and upload a pic with jpg extension\")\n",
        "  os.remove(\"./\" + pic_name)\n",
        "else:\n",
        "  shutil.move(\"./\" + pic_name, \"/content/3d-photo-inpainting-master/image/\" + pic_name)"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-e8a91b31-4fbb-442d-80aa-655acf2bb9c6\" name=\"files[]\" multiple disabled />\n",
              "     <output id=\"result-e8a91b31-4fbb-442d-80aa-655acf2bb9c6\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving doublestrike.jpg to doublestrike.jpg\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IYC4c6r6vjVh",
        "colab_type": "text"
      },
      "source": [
        "## 4. Execute the 3D Photo Inpainting\n",
        "  - Note: The 3D photo generation process usually takes about 2-3 minutes or more depending on the available computing resources."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zo5uPQuBgw2j",
        "colab_type": "code",
        "outputId": "418a0e95-ee06-4497-aa68-304b4c98774e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 454
        }
      },
      "source": [
        "!python main.py --config argument.yml"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\r  0% 0/1 [00:00<?, ?it/s]Current Source ==>  doublestrike\n",
            "initialize\n",
            "device: cpu\n",
            "start processing\n",
            "  processing image/doublestrike.jpg (1/1)\n",
            "torch.Size([1, 3, 288, 384])\n",
            "finished\n",
            "Start Running 3D_Photo ...\n",
            "Writing mesh file mesh/doublestrike.ply ...\n",
            "53.13010235415598\n",
            "Moviepy - Building video video/doublestrike_straight-line.mp4.\n",
            "Moviepy - Writing video video/doublestrike_straight-line.mp4\n",
            "\n",
            "  0% 0/1 [02:36<?, ?it/s]\n",
            "t:   0% 0/240 [00:00<?, ?it/s, now=None]\u001b[A\n",
            "t:   1% 2/240 [00:00<00:13, 18.28it/s, now=None]\u001b[A\n",
            "t:   2% 4/240 [00:00<00:13, 17.07it/s, now=None]\u001b[A\n",
            "t:   8% 18/240 [00:00<00:09, 23.16it/s, now=None]\u001b[A\n",
            "t:  15% 35/240 [00:00<00:06, 31.18it/s, now=None]\u001b[A\n",
            "t:  20% 47/240 [00:00<00:04, 39.87it/s, now=None]\u001b[A\n",
            "t:  24% 57/240 [00:00<00:04, 37.29it/s, now=None]\u001b[A\n",
            "t:  27% 65/240 [00:01<00:04, 38.59it/s, now=None]\u001b[A\n",
            "t:  30% 72/240 [00:01<00:04, 39.76it/s, now=None]\u001b[A\n",
            "t:  32% 78/240 [00:01<00:03, 40.65it/s, now=None]\u001b[A\n",
            "t:  35% 84/240 [00:01<00:03, 39.81it/s, now=None]\u001b[A"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wPvkMT0msIJB",
        "colab_type": "text"
      },
      "source": [
        "- **The results are stored in the following directories**\n",
        "  - Corresponding depth map estimated by [MiDaS](https://github.com/intel-isl/MiDaS.git) \n",
        "      - E.g. ```/content/3d-photo-inpainting-master/depth/moon.npy```\n",
        "  - Inpainted 3D mesh\n",
        "      - E.g. ```/content/3d-photo-inpainting-master/mesh/moon.ply```\n",
        "  - Rendered videos with circular motion\n",
        "      - E.g. ```/content/3d-photo-inpainting-master/mesh/moon.mp4```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r8yS0VGK2Fw7",
        "colab_type": "text"
      },
      "source": [
        "## 5. Visualize results\n",
        "### 5.1 Swing effect"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ONYMrVBlzPBm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 471
        },
        "outputId": "6eb1b76f-b967-40bb-9ae4-b4e4bcc0064d"
      },
      "source": [
        "from IPython.display import HTML\n",
        "from base64 import b64encode\n",
        "mp4 = open('/content/3d-photo-inpainting-master/video/' + pic_name.split(\".\")[0] +  '_swing.mp4','rb').read()\n",
        "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
        "HTML(\"\"\"\n",
        "<video width=600 controls autoplay>\n",
        "      <source src=\"%s\" type=\"video/mp4\">\n",
        "</video>\n",
        "\"\"\" % data_url)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "<video width=600 controls autoplay>\n",
              "      <source src=\"data:video/mp4;base64,[encoded data, 646740 characters]\" type=\"video/mp4\">\n",
              "</video>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PkWOHYuT2WU2",
        "colab_type": "text"
      },
      "source": [
        "### 5.2 Straight line effect"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OpZtMhey1n1W",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 471
        },
        "outputId": "68f7b379-2020-4d72-cbe0-3d64f06c23e1"
      },
      "source": [
        "mp4 = open('/content/3d-photo-inpainting-master/video/' + pic_name.split(\".\")[0] +  '_straight-line.mp4','rb').read()\n",
        "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
        "HTML(\"\"\"\n",
        "<video width=600 controls autoplay>\n",
        "      <source src=\"%s\" type=\"video/mp4\">\n",
        "</video>\n",
        "\"\"\" % data_url)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "<video width=600 controls autoplay>\n",
              "      <source src=\"data:video/mp4;base64,[encoded data, 483284 characters]\" type=\"video/mp4\">\n",
              "</video>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N3kre-V32c97",
        "colab_type": "text"
      },
      "source": [
        "## Download it!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lck4NL4y2iXb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "files.download('/content/3d-photo-inpainting-master/video/' + pic_name.split(\".\")[0] +  '_swing.mp4')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hx8M7tPg2ysr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "files.download('/content/3d-photo-inpainting-master/video/' + pic_name.split(\".\")[0] +  '_straight-line.mp4')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2OLBQstJ3ZwK",
        "colab_type": "text"
      },
      "source": [
        "### Download 3D generated file (PLY)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9Y3pSoKB3fBL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "files.download('/content/3d-photo-inpainting-master/mesh/' + + pic_name.split(\".\")[0] + '.ply')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zbPquetz7_AZ",
        "colab_type": "text"
      },
      "source": [
        "### If you want to try with another picture go to step 3"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_tYHvK2i34pV",
        "colab_type": "text"
      },
      "source": [
        "<p>Did you enjoy? Buy me a coffe :)</p>\n",
        "<p><a href=\"https://ko-fi.com/Y8Y3VYYE\"><img src=\"https://camo.githubusercontent.com/c8a9d4f1653d599167ef09852550c6810a7306bc/68747470733a2f2f7777772e6b6f2d66692e636f6d2f696d672f676974687562627574746f6e5f736d2e737667\" alt=\"ko-fi\" data-canonical-src=\"https://www.ko-fi.com/img/githubbutton_sm.svg\"></a></p>"
      ]
    }
  ]
}