Home

Awesome

<p align="center"> <img src="assets/logo_color.png" height=180> </p>

😴 LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes 😴

<div align="center">

Project ArXiv Open In Colab Open in Spaces

</div> <div align="center">

Github X LICENSE

</div>

https://github.com/luciddreamer-cvlab/LucidDreamer/assets/12259041/35004aaa-dffc-4133-b15a-05224e68b91e

LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes

*Jaeyoung Chung, *Suyoung Lee, Hyeongjin Nam, Jaerin Lee, Kyoung Mu Lee
*Denotes equal contribution.
<p align="center"> <img src="assets/logo_cvlab.png" height=60> </p>

🤖 Install

Ubuntu

Prerequisite

Installation script

conda create -n lucid python=3.9
conda activate lucid
pip install peft diffusers scipy numpy imageio[ffmpeg] opencv-python Pillow open3d torch==2.0.1  torchvision==0.15.2 gradio omegaconf
# ZoeDepth
pip install timm==0.6.7
# Gaussian splatting
pip install plyfile==0.8.1

cd submodules/depth-diff-gaussian-rasterization-min
# sudo apt-get install libglm-dev # may be required for the compilation.
python setup.py install
cd ../simple-knn
python setup.py install
cd ../..

Windows (Experimental, Tested on Windows 11 with VS2022)

Checklist

Installation script

conda create -n lucid python=3.9
conda activate lucid
conda install pytorch=2.0.1 torchvision=0.15.2 torchaudio=2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install peft diffusers scipy numpy imageio[ffmpeg] opencv-python Pillow open3d gradio omegaconf
# ZoeDepth
pip install timm==0.6.7
# Gaussian splatting
pip install plyfile==0.8.1

# There is an issue with whl file so please manually install the module now.
cd submodules\depth-diff-gaussian-rasterization-min\third_party
git clone https://github.com/g-truc/glm.git
cd ..\
python setup.py install
cd ..\simple-knn
python setup.py install
cd ..\..

⚡ Usage

We offer several ways to interact with LucidDreamer:

  1. A demo is available on ironjr/LucidDreamer HuggingFace Space (including custom SD ckpt) and ironjr/LucidDreamer-mini HuggingFace Space (minimal features / try at here in case of the former is down) (We appreciate all the HF / Gradio team for their support).

https://github.com/luciddreamer-cvlab/LucidDreamer/assets/12259041/745bfc46-8215-4db2-80d5-4825e91316bc

  1. Another demo is available on a Colab, implemented by @camenduru (We greatly thank @camenduru for the contribution).
  2. You can use the gradio demo locally by running CUDA_VISIBLE_DEVICES=0 python app.py (full feature including huggingface model download, requires ~15GB) or CUDA_VISIBLE_DEVICES=0 python app_mini.py (minimum viable demo, uses only SD1.5).
  3. You can also run this with command line interface as described below.

Run with your own samples

# Default Example
python run.py --image <path_to_image> --text <path_to_text_file> [Other options]

Other options

Guideline for the prompting / Troubleshoot

General guides

  1. If your image is indoors with specific scene (and possible character in it), you can just put the most simplest representation of the scene first, like a cozy livingroom for christmas, or a dark garage, etc. Please avoid prompts like 1girl because it will generate many humans for each inpainting task.
  2. If you want to start from already hard-engineered image from e.g., StableDiffusion model, or a photo taken from other sources, you can try using WD14 tagger (huggingface demo) to extract the danbooru tags from an image. Please ensure you remove some comma separated tags if you don't want them to appear multiple times. This include human-related objects, e.g., 1girl, white shirt, boots, smiling face, red eyes, etc. Make sure to specify the objects you want to have multiples of them.

Q. I generate unwanted objects everywhere, e.g., photo frames.

  1. Manipulate negative prompts to set harder constraints for the frame object. You may try adding tags like twitter thumbnail, profile image, instagram image, watermark, text to the negative prompt. In fact, negative prompts are the best thing to try if you want some things not to be appeared in the resulting image.
  2. Try using other custom checkpoint models, which employs different pipeline methods: LaMa inpainting -> ControlNet-inpaint guided image inpainting.

Visualize .ply files

There are multiple available viewers / editors for Gaussian splatting .ply files.

  1. @playcanvas's Super-Splat project (Live demo). This is the viewer we have used for our debugging along with MeshLab.

image

  1. @antimatter15's WebGL viewer for Gaussian splatting (Live demo).

  2. @splinetool's web-based viewer for Gaussian splatting. This is the version we have used in our project page's demo.

🚩 Updates

🌏 Citation

Please cite us if you find our project useful!

@article{chung2023luciddreamer,
    title={LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes},
    author={Chung, Jaeyoung and Lee, Suyoung and Nam, Hyeongjin and Lee, Jaerin and Lee, Kyoung Mu},
    journal={arXiv preprint arXiv:2311.13384},
    year={2023}
}

🤗 Acknowledgement

We deeply appreciate ZoeDepth, Stability AI, and Runway for their models.

📧 Contact

If you have any questions, please email robot0321@snu.ac.kr, esw0116@snu.ac.kr, jarin.lee@gmail.com.

⭐ Star History

<a href="https://star-history.com/#luciddreamer-cvlab/LucidDreamer&Date"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=luciddreamer-cvlab/LucidDreamer&type=Date&theme=dark" /> <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=luciddreamer-cvlab/LucidDreamer&type=Date" /> <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=luciddreamer-cvlab/LucidDreamer&type=Date" /> </picture> </a>