Awesome
(ECCV 2024) Learn to Optimize Denoising Scores: A Unified and Improved Diffusion Prior for 3D Generation
Project Page | Arxiv Paper
<p style='text-align: justify;'> In this paper, we propose a unified framework aimed at enhancing the diffusion priors for 3D generation tasks. Despite the critical importance of these tasks, existing methodologies often struggle to generate high-caliber results. We begin by examining the inherent limitations in previous diffusion priors. We identify a divergence between the diffusion priors and the training procedures of diffusion models that substantially impairs the quality of 3D generation. To address this issue, we propose a novel, unified framework that iteratively optimizes both the 3D model and the diffusion prior. Leveraging the different learnable parameters of the diffusion prior, our approach offers multiple configurations, affording various trade-offs between performance and implementation complexity. Notably, our experimental results demonstrate that our method markedly surpasses existing techniques, establishing new state-of-the-art in the realm of text-to-3D generation. Additionally, our framework yields insightful contributions to the understanding of recent score distillation methods, such as the VSD loss and CSD loss. </p>🔥🔥NEW
Interested in Stable Diffusion 3 to 3D? Check out our new work here.
Installation
- Install pytorch and torch vision
# torch1.12.1+cu113
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
# or torch2.0.0+cu118
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
- Install glm (required by 3D Gaussian Splatting)
sudo apt-get install -y libglm-dev
- Install dependencies:
pip install -r requirements.txt
- Install 3DGS and Shap-E:
pip install ninja
pip install ./gaussiansplatting/submodules/diff-gaussian-rasterization
pip install ./gaussiansplatting/submodules/simple-knn
git clone https://github.com/openai/shap-e.git
cd shap-e
pip install -e .
Our codes are based on the implementations of ThreeStudio and GaussianDreamer. If you have any problem with the installation, you may search the issues in these two repos first. Also feel free to open a new issue here.
Quickstart
2D Playground
python 2dplayground_lora.py
python 2dplayground_embedding.py
You should get results similar to these:
LoRA | Embedding |
---|---|
Run LODS Embedding + 3D Gaussian Splatting
python launch.py --config configs/lods-gs-embedding.yaml --train --gpu 0 system.prompt_processor.prompt="a DSLR image of a hamburger"
Run LODS LoRA + Instant-NGP
python launch.py --config configs/lods-ngp-lora.yaml --train --gpu 0 system.prompt_processor.prompt="a DSLR image of a hamburger"
Run LODS Embedding + Instant-NGP
python launch.py --config configs/lods-ngp-embedding.yaml --train --gpu 0 system.prompt_processor.prompt="a DSLR image of a hamburger"
Credits
LODS is built on the following open-source projects:
- ThreeStudio Main Framework
- GaussianDreamer 3D Gaussian Splatting
- 3D Gaussian Splatting 3D Gaussian Splatting
Credits from ThreeStudio
- Lightning Framework for creating highly organized PyTorch code.
- OmegaConf Flexible Python configuration system.
- NerfAcc Plug-and-play NeRF acceleration.
Citation
@article{yang2024lods,
title={Learn to Optimize Denoising Scores: A Unified and Improved Diffusion Prior for 3D Generation},
author={Xiaofeng Yang and Yiwen Chen and Cheng Chen and Chi Zhang and Yi Xu and Xulei Yang and Fayao Liu and Guosheng Lin},
journal={ECCV 2024},
year={2024}
}