Home

Awesome

High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization (CVPR 2023)

paper | project website

<img src="pics/input02.png" width="200px"/> <img src="pics/input_video02.gif" width="200"/> <img src="pics/input07.png" width="200px"/> <img src="pics/input_video07.gif" width="200"/>

<img src="pics/input03.png" width="200px"/> <img src="pics/input_video03.gif" width="200"/> <img src="pics/input04.png" width="200px"/> <img src="pics/input_video04.gif" width="200"/>

Introduction

We present a high-fidelity 3D generative adversarial net- work (GAN) inversion framework that can synthesize photo- realistic novel views while preserving specific details of the input image.

<img src="pics/method.png" width="800px"/>

Set up

Installation

git clone https://github.com/jiaxinxie97/HFGI3D.git
cd HFGI3D

Environment

conda env create -f environment.yml
conda activate HFGI3D
pip install -r pose_requirements.txt

To Do

Quick Start

Prepare Images

We put some examples images and their corresponding pose in ./test_imgs, also we put the configs files of examples in ./example_configs/, and you can quickly try them.

For customized images, download the pretrained pose estimation model and BFM files, put epoch_20.pth in ./pose_estimation/checkpoints/pretrained/ and put unzip BFM.zip in ./pose_estimation/

cd inversion/pose_estimation
python extract_pose.py 0 custom_imgs_folder temp_folder output_folder

###example
#python extract_pose.py 0 ../custom_imgs ../align_imgs ../test_imgs

Then create config files for customized images in ./example_configs/

Pretraind model

Download the pretrained generator on FFHQ from EG3D. For convenience, we upload it in Google drive(https://drive.google.com/file/d/1rsF-IHBLW7WvDckdbNK9Qm9SwHK02E5l/view?usp=sharing). Download ffhq512-128.pkl and put it in ./inversion/.

Optimization

cd inversion/scripts
python run_pti.py ../../example_configs/config_00001.py

For customized images,

cd inversion/scripts
python run_pti.py ../../example_configs/customized_config.py

More Results

Video results are shown on our project website (https://ken-ouyang.github.io/HFGI3D/index.html).

Acknowlegement

We thank the authors of EG3D, PTI and Deep3DFaceRecon for sharing their code.

Citation

If you find this work useful for your research, please cite:

@article{xie2022high,
  title={High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization},
  author={Xie, Jiaxin and Ouyang, Hao and Piao, Jingtan and Lei, Chenyang and Chen, Qifeng},
  journal={arXiv preprint arXiv:2211.15662},
  year={2022}
}