Awesome

DreamControl

Implementation of CVPR 2024 paper "DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior"

Installation

We build our source code based on original threestudio. See installation.md for additional information.

Qucik Start

We provide two ways to load shape conditions for our control-based score distillation.

# load an obj file as a condition:
python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="{your text prompt}" system.geometry_c.shape_init="mesh:{path of the obj file}"

# load a threestudio checkpoint as a condition:
python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="{your text prompt}" system.geometry_c.shape_init="{path of the last checkpoint file}"

For example, you can use the template geometry we provide in ./load/rot_shapes or pre-process a new shape with correct orientation (Note that, the orientation of original geoemtries in threestudio's ./load/shapes is different from ours):

# rotate the geometry
cd ./load/rot_shapes
python rot.py ../shapes/nascar.obj

# optimize a NeRF with DreamControl
cd ../..
python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="A Tesla Model3 sedan" system.geometry_c.shape_init="mesh:./load/rot_shapes/nascar.obj"

Or, you can use a threestudio checkpoint. We provide an example here. You can download and unzip it, and then:

python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="a corgi" system.geometry_c.shape_init="./a_corgi/ckpts/last.ckpt"

TODO

• Code of Control-Based Score Distribution.
• Code of 3D Self-Prior Generation.

Gallery

Acknowledgement

Our codes are built on threestudio, and ControlLoRA.

Citation

@article{huang2023dreamcontrol,
  title={Dreamcontrol: Control-based text-to-3d generation with 3d self-prior},
  author={Huang, Tianyu and Zeng, Yihan and Zhang, Zhilu and Xu, Wan and Xu, Hang and Xu, Songcen and Lau, Rynson WH and Zuo, Wangmeng},
  journal={arXiv preprint arXiv:2312.06439},
  year={2023}
}