Awesome
DreamView
Official PyTorch implementation of ECCV 2024 paper “DreamView: Injecting View-specific Text Guidance into Text-to-3D Generation”.
This repository includes the training and inference code for text-to-image generation (DreamView-2D), and text-to-3D generation (DreamView-3D).
You can check DreamView-2D/
and DreamView-3D/
for the text-to-image and text-to-3D generation codes, respectively.
News
- 2024/7/22: We release the code and script for training DreamView-2D and text-to-3D generation via DreamView-3D
- 2024/7/3: Our paper is accepted by ECCV 2024, congratulations and many thanks to the co-authors!🥳🎉🎊
- 2024/4/11: We release the inference code of DreamView-2D, and the DreamView checkpoint: Baidu Drive, Google Drive, ModelScope, HuggingFace
- 2024/4/10: Our arxiv paper can be found here
TODO
- Release the code of DreamView-3D
- Release the training script for training DreamView-2D
Preliminary
Environment
- For DreamView-2D: please follow MVDream to prepare the environments
- For DreamView-3D: please follow threestudio to prepare the environments
Checkpoint
- Download the checkpoints at Baidu Drive, Google Drive, ModelScope, HuggingFace
- Move the checkpoints to
ckpts/
Text-to-image Generation
Inference
Running the below script
cd DreamView-2D
python t2i.py --num_samples 4 --fp16
and you are expected to obtain the below result:
It takes about 10G GPU memory to run the text-to-image generation, and you can modify the DreamView-2D/t2i.py
to generate your own content.
Training
cd DreamView-2D
bash train.sh
Note that we use 4 8*V100 machine to train DreamView-2D by default, and to accelerate convergence, you can consider using MVDream as the initialization parameter.
Text-to-3D Generation
Running the script below to reproduce the results shown above
cd DreamView-3D
bash reproduce.sh
Note that the above script may require ~60G GPU memory, so you may run it with an A100 GPU.
Acknowledgement
- The code of DreamView-2d is heavily based on MVDream and Stable Diffusion.
- The code of DreamView-3d is heavily based on MVDream-threestudio and threestudio.
- We would like to thank the authors for sharing their exciting works.
Citation
- If you find this repo helpful, please consider citing us:
@inproceedings{yan2024DreamView,
author = {Yan, Junkai and Gao, Yipeng and Yang, Qize and Wei, Xihan and Xie, Xuansong and Wu, Ancong and Zheng, Wei-Shi},
title = {DreamView: Injecting View-specific Text Guidance into Text-to-3D Generation},
booktitle = {ECCV},
year = {2024}
}