Awesome
GVGEN: Text-to-3D Generation with Volumetric Representation 🧊
<div align="center"> </div> <p align="center"> <img src="assets/gvgen_teaser.gif" width=95%> <p>🔥 Update
- [2024.10.16] Code for GaussianVolume Fitting is released ! See CPS !
- [2024.07.04] Code and Models for text-conditional 3D generation are released !
- [2024.07.04] GVGEN was accepted by ECCV 2024. See you in Milan!
🌿 Introduction
We introduce GVGEN, a novel diffusion-based framework, which is designed to efficiently generate 3D Gaussian representations from text input. We propose two innovative techniques:
Structured Volumetric Representation
. We first arrange disorganized 3D Gaussian points as a structured form GaussianVolume. This transformation allows the capture of intricate texture details within a volume composed of a fixed number of Gaussians.Coarse-to-fine Generation Pipeline
. To simplify the generation of GaussianVolume and empower the model to generate instances with detailed 3D geometry, we propose a coarse-to-fine pipeline. It initially constructs a basic geometric structure, followed by the prediction of complete Gaussian attributes.
🦄 Text-conditional 3D generation
Environment Setup
conda create -n gvgen python=3.8
pip install -r requirements.txt
Then, install the diff-gaussian-rasterization
submodule according to the instructions provided by 3DGS
Pretrained Models
Please download models from , put them in the folder ./ckpts
.
Run
After completing all the above instructions, run
python run_text.py --text_input YOUR_TEXT_INPUT
# for example
python run_text.py --text_input "a green truck"
The generated gif and 3DGS will be saved to sample.gif
and sample.ply
, respectively.
The text condition we used during training is derived from Cap3D.
We recommend everyone to imitate the style of Cap3D's text and create your own prompts for better generation results.
⚡️ ToDo List
-
Release Code for GaussianVolume fitting -
Release Code for data preprocessing
-
Release Code for training
License
The majority of this project is licensed under MIT License. Portions of the project are available under separate license of referred projects, detailed in corresponding files.
BibTeX
@article{he2024gvgen,
title={GVGEN: Text-to-3D Generation with Volumetric Representation},
author={He, Xianglong and Chen, Junyi and Peng, Sida and Huang, Di and Li, Yangguang and Huang, Xiaoshui and Yuan, Chun and Ouyang, Wanli and He, Tong},
journal={arXiv preprint arXiv:2403.12957},
year={2024}
}