# 3D-Adapter

Official PyTorch implementation of the papers:

**3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation**
<br>
[Hansheng Chen](https://lakonik.github.io/)<sup>1</sup>, 
[Bokui Shen](https://cs.stanford.edu/people/bshen88/)<sup>2</sup>,
[Yulin Liu](https://liuyulinn.github.io/)<sup>3,4</sup>,
[Ruoxi Shi](https://rshi.top/)<sup>3</sup>, 
[Linqi Zhou](https://alexzhou907.github.io/)<sup>2</sup>,
[Connor Z. Lin](https://connorzlin.com/)<sup>2</sup>,
[Jiayuan Gu](https://pages.ucsd.edu/~ztu/)<sup>3</sup>, 
[Hao Su](https://cseweb.ucsd.edu/~haosu/)<sup>3,4</sup>,
[Gordon Wetzstein](http://web.stanford.edu/~gordonwz/)<sup>1</sup>, 
[Leonidas Guibas](https://geometry.stanford.edu/member/guibas/)<sup>1</sup><br>
<sup>1</sup>Stanford University, <sup>2</sup>Apparate Labs, <sup>3</sup>UCSD, <sup>4</sup>Hillbot
<br>
[[Project page](https://lakonik.github.io/3d-adapter)] [[🤗Demo](https://huggingface.co/spaces/Lakonik/3D-Adapter)] [[Paper](https://arxiv.org/abs/2410.18974)]

**Generic 3D Diffusion Adapter Using Controlled Multi-View Editing**
<br>
[Hansheng Chen](https://lakonik.github.io/)<sup>1</sup>, 
[Ruoxi Shi](https://rshi.top/)<sup>2</sup>, 
[Yulin Liu](https://liuyulinn.github.io/)<sup>2</sup>, 
[Bokui Shen](https://cs.stanford.edu/people/bshen88/)<sup>3</sup>,
[Jiayuan Gu](https://pages.ucsd.edu/~ztu/)<sup>2</sup>, 
[Gordon Wetzstein](http://web.stanford.edu/~gordonwz/)<sup>1</sup>, 
[Hao Su](https://cseweb.ucsd.edu/~haosu/)<sup>2</sup>, 
[Leonidas Guibas](https://geometry.stanford.edu/member/guibas/)<sup>1</sup><br>
<sup>1</sup>Stanford University, <sup>2</sup>UCSD, <sup>3</sup>Apparate Labs
<br>
[[Project page](https://lakonik.github.io/mvedit)] [[🤗Demo](https://huggingface.co/spaces/Lakonik/MVEdit)] [[Paper](https://arxiv.org/abs/2403.12032)]


https://github.com/user-attachments/assets/6cba3a92-04fe-46ee-88ca-e6dfe5443c36


## Todos

- [ ] Release GRM-based 3D-Adapters (unfortunately, we cannot release these models before the official release of [GRM](https://github.com/justimyhxu/GRM))

## Installation

The code has been tested in the environment described as follows:

- Linux (tested on Ubuntu 20 and above)
- [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit-archive) 11.8 and above
- [PyTorch](https://pytorch.org/get-started/previous-versions/) 2.1 and above
- FFmpeg, x264 (optional, for exporting videos)

Other dependencies can be installed via `pip install -r requirements.txt`. 

An example of installation commands is shown below (you may change the CUDA version yourself):

```bash
# Export the PATH of CUDA toolkit
export PATH=/usr/local/cuda-12.1/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH

# Create conda environment
conda create -y -n mvedit python=3.10
conda activate mvedit

# Install FFmpeg (optional)
conda install -c conda-forge ffmpeg x264

# Install PyTorch
conda install pytorch==2.1.2 torchvision==0.16.2 pytorch-cuda=12.1 -c pytorch -c nvidia

# Clone this repo and install other dependencies
git clone https://github.com/Lakonik/MVEdit && cd MVEdit
pip install -r requirements.txt
```

This codebase also works on Windows systems if the environment is configured correctly. Please refer to [Issue #8](https://github.com/Lakonik/MVEdit/issues/8) for more information about the environment setup on Windows.

## Inference

We recommend using the Gradio Web UI and its APIs for inference. A GPU with at least 24GB of VRAM is required to run the Web UI.

### Web UI

Run the following command to start the Web UI:

```bash
python app.py --unload-models
```

The Web UI will be available at [http://localhost:7860](http://localhost:7860). If you add the `--share` flag, a temporary public URL will be generated for you to share the Web UI with others.

All models will be automatically loaded on demand. The first run will take a very long time to download the models. Check your network connection to GitHub, Google Drive and Hugging Face if the download fails.

To view other options, run:

```bash
python app.py -h
```

### API

After starting the Web UI, the API docs will be available at [http://localhost:7860/?view=api](http://localhost:7860/?view=api). The docs are automatically generated by Gradio, and the data types and default values may be incorrect. Please use the default values in the Web UI as a reference.

**Please refer to our [examples](scripts/) for API usage with python.**

## Training

Optimization-based 3D-Adapters (a.k.a. MVEdit adapters) adopt only off-the-shelf models and require no further training.

The training code for GRM-based 3D-Adapters will be released after the official release of [GRM](https://github.com/justimyhxu/GRM). 

## Acknowledgements

This codebase is built upon the following repositories:
- Base library modified from [SSDNeRF](https://github.com/Lakonik/SSDNeRF)
- NeRF renderer and DMTet modified from [Stable-DreamFusion](https://github.com/ashawkey/stable-dreamfusion)
- Gaussian Splatting renderer modified from [3DGS](https://github.com/graphdeco-inria/gaussian-splatting) and [Differential Gaussian Rasterization](https://github.com/ashawkey/diff-gaussian-rasterization)
- Mesh I/O modified from [DreamGaussian](https://github.com/dreamgaussian/dreamgaussian)
- [GRM](https://github.com/justimyhxu/GRM) for Gaussian reconstruction
- [Zero123++](https://github.com/SUDO-AI-3D/zero123plus) for image-to-3D initialization
- [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter) for extra conditioning
- [TRACER](https://github.com/Karel911/TRACER) for background removal
- [LoFTR](https://github.com/zju3dv/LoFTR) for pose estimation in image-to-3D
- [Omnidata](https://github.com/EPFL-VILAB/omnidata) for normal prediction in image-to-3D
- [Image Packer](https://github.com/theFroh/imagepacker) for mesh preprocessing

## Citation

```bibtex
@misc{3dadapter2024,
    title={3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation},
    author={Hansheng Chen and Bokui Shen and Yulin Liu and Ruoxi Shi and Linqi Zhou and Connor Z. Lin and Jiayuan Gu and Hao Su and Gordon Wetzstein and Leonidas Guibas},
    year={2024},
    eprint={2410.18974},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2410.18974},
}

@misc{mvedit2024,
    title={Generic 3D Diffusion Adapter Using Controlled Multi-View Editing},
    author={Hansheng Chen and Ruoxi Shi and Yulin Liu and Bokui Shen and Jiayuan Gu and Gordon Wetzstein and Hao Su and Leonidas Guibas},
    year={2024},
    eprint={2403.12032},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}
```
