Home

Awesome

Region-Based Semantic Factorization in GANs

image

Figure: Image editing results using ReSeFa on BigGAN (first two columns) and StyleGAN2 (last three columns).

Region-Based Semantic Factorization in GANs <br> Jiapeng Zhu, Yujun Shen, Yinghao Xu, Deli Zhao, Qifeng Chen <br> International Conference on Machine Learning (ICML) 2022 <br>

[Paper] [Project page] [Demo] [Colab]

In the repository, we propose a simple algorithm to interpret the region-based semantics learned by GANs. In particular, we re-examine the task of local editing with pre-trained GAN models, and formulate region-based semantic discovery as a dual optimization problem. Through an appropriately defined generalized Rayleigh quotient, we are able to tackle such a problem super efficiently with a closed-form solution. Extensive experiments on BigGAN and StyleGAN2 demonstrate the effectiveness and robustness of our proposed method.

Local Face Editing Results

image

Figure: Semantics found by ReSeFa with respect to the region of interest (i.e., within the green boxes).

Usage

Pre-trained Models

This repository is based on Hammer, where you can find detailed instructions on environmental setup. Please refer here for model downloading and conversion, and use the script synthesis.py to check whether the model is ready for use as

MODEL_PATH='stylegan2-ffhq-config-f-1024x1024.pth'
python synthesis.py ${MODEL_PATH} --stylegan2

Editing with Provided Directions

We have provided some local semantic directions under the directory directions/, which are discovered from the officially released StyleGAN2/StyleGAN3 models. Users can use these directions for image local editing. For example,

Finding Your Own Directions

To explore more directions, please follow the following steps, which use the FFHQ model officially released in StyleGAN2 as an example.

BibTeX

 @inproceedings{zhu2022resefa,
  title     = {Region-Based Semantic Factorization in {GAN}s},
  author    = {Zhu, Jiapeng and Shen, Yujun and Xu, Yinghao and Zhao, Deli and Chen, Qifeng},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2022}
}