Home

Awesome

DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling (CVPR 2024 Highlight)

Official Implementation of DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling

<p align="center"> <img src="assets/teaser.gif" width="90%" loop=infinite/> </p> <div align="center"> <span class="author-block"> <a href="https://alexzhou907.github.io/">Linqi Zhou</a><sup>1</sup>,</span> <span class="author-block"> <a href="https://cs.stanford.edu/~andyshih/">Andy Shih</a><sup>1</sup>,</span> <span class="author-block"> <a href="https://cs.stanford.edu/~chenlin/">Chenlin Meng</a><sup>1,2</sup>, </span> <span class="author-block"> <a href="https://cs.stanford.edu/~ermon/">Stefano Ermon</a><sup>1</sup> </span> </div> <div align="center"> <span class="author-block"><sup>1</sup>Stanford University,</span> <span class="author-block"><sup>2</sup>Pika Labs</span> </div> <div align="center"> <a href="https://arxiv.org/abs/2311.17082">[Paper]</a> <a href="https://alexzhou907.github.io/dreampropeller_page/">[Project]</a> </div>

This codebase is based on threestudio with minor changes to some files for parallel processing.

Installation

conda create -n dreampropeller python=3.11
pip install ninja
pip install -r requirements.txt
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121

Usage

We can compare the runtime for baseline and DreamPropeller by using launch_baseline.py and launch_speedup.py respectively. For DreamPropeller, one can tune speedup parameters in .yaml config files. The preset values in the files are the default values. We note their meaning below:

Other visualization options:

We can quickly test each of the frameworks presented in the paper below. Our method is tested on an 8-GPU A100-80G PCIe cluster.

You can freely switch between baseline and DreamPropeller by simply setting FILE to launch_baseline.py or launch_speedup.py.

DreamFusion

python $FILE --config configs/dreamfusion-if.yaml --train system.prompt_processor.prompt="an ice cream sundae" data.batch_size=16 trainer.visualize_progress=true

Magic3D

python $FILE  --config configs/magic3d-coarse-if.yaml --train system.prompt_processor.prompt="a beautiful dress made out of fruit, on a mannequin. Studio lighting, high quality, high resolution" data.batch_size=16 trainer.visualize_progress=true

python $FILE --config configs/magic3d-refine-sd.yaml --train system.prompt_processor.prompt="a beautiful dress made out of fruit, on a mannequin. Studio lighting, high quality, high resolution" data.batch_size=16 trainer.visualize_progress=true system.geometry_convert_from=path/to/coarse/stage/trial/dir/ckpts/last.ckpt

TextMesh

python $FILE --config configs/textmesh-if.yaml --train system.prompt_processor.prompt="an old vintage car" data.batch_size=16 trainer.visualize_progress=true

ProlificDreamer

python $FILE --config configs/prolificdreamer.yaml --train system.prompt_processor.prompt="a detailed Victorian era house" data.batch_size=[8,2] trainer.visualize_progress=true

python $FILE --config configs/prolificdreamer-geometry.yaml --train system.prompt_processor.prompt="a detailed Victorian era house" data.batch_size=8 trainer.visualize_progress=true system.geometry_convert_from=path/to/coarse/stage/trial/dir/ckpts/last.ckpt

python $FILE --config configs/prolificdreamer-texture.yaml --train system.prompt_processor.prompt="a detailed Victorian era house"  data.batch_size=1 trainer.visualize_progress=true system.geometry_convert_from=path/to/geometry/stage/trial/dir/ckpts/last.ckpt

DreamGaussian

Since DreamGaussian is a separate standalone repo, we have copied it in the dreamgaussian/ subfolder with our dreampropeller implementation specifically tailored to this repo. Please find specific instructions to run DreamGaussian inside.

Zero 1-to-3

Our method can also be applied to Image-to-3D using score distillation. Luckily Zero 1-to-3 is implemented in the original threestudio code, so we can directly plug in our method.

Download pretrained Zero123XL weights into load/zero123:

cd load/zero123
wget https://zero123.cs.columbia.edu/assets/zero123-xl.ckpt

To run,

python $FILE --config configs/zero123.yaml --train data.image_path=./load/images/grootplant_rgba.png data.random_camera.batch_size=[16,16,10] trainer.visualize_progress=true

Reference

@article{zhou2023dreampropeller,
  title={DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling},
  author={Zhou, Linqi and Shih, Andy and Meng, Chenlin and Ermon, Stefano},
  journal={arXiv preprint arXiv:2311.17082},
  year={2023}
}