Home

Awesome

StyleGAN2: projecting images

The goal of this Google Colab notebook is to project images to latent space with StyleGAN2.

Usage

To discover how to project a real image using the original StyleGAN2 implementation, run:

To process the projection of a batch of images, using either W(1,*) (original) or W(18,*) (extended), run:

To edit latent vectors of projected images, run:

For more information about W(1,*) and W(18,*), please refer to the the original paper (section 5 on page 7):

Inverting the synthesis network $g$ is an interesting problem that has many applications. Manipulating a given image in the latent feature space requires finding a matching latent code $w$ for it first.

The following is about W(18,*):

Previous research suggests that instead of finding a common latent code $w$, the results improve if a separate $w$ is chosen for each layer of the generator. The same approach was used in an early encoder implementation.

The following is about W(1,*), which is the approach used in the original implementation:

While extending the latent space in this fashion finds a closer match to a given image, it also enables projecting arbitrary images that should have no latent representation. Instead, we concentrate on finding latent codes in the original, unextended latent space, as these correspond to images that the generator could have produced.

Data

Data consists of:

<img alt="Original image of the French president" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/img/emmanuel-macron.jpg" width="250">

Pre-processing

There are two possible pre-processing methods:

<img alt="Center-cropping" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/img/emmanuel-macron-crop.jpg" width="250"> <img alt="Face landmarks" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/face_landmarks/landmarks_macron.jpg" width="250"> <img alt="FFHQ pre-processing" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/img/emmanuel-macron-aligned.jpg" width="250">

Finally, the pre-processed image can be projected to the latent space of the StyleGAN2 model trained with configuration f on the Flickr-Faces-HQ (FFHQ) dataset.

Results: influence of pre-processing

NB: results are different if the code is run twice, even if the same pre-processing is used.

With center-cropping as sole pre-processing

The result below is obtained with center-cropping as sole pre-processing, hence some issues with the projection.

Projection (with issues) as GIF

From left to right: the target image, the result obtained at the start of the projection, and the final result of the projection.

<img alt="Target image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/00000-project-real-images/image0000-target.jpg" width="250"><img alt="Projected image n°1/5" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/00000-project-real-images/image0000-step0200.jpg" width="250"><img alt="Projected image n°5/5" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/00000-project-real-images/image0000-step1000.jpg" width="250">

From left to right: the target image, the result obtained at the start of the projection, intermediate results, and the final result.

Projection results (with issues) as PNG

The background, the hair, the ears, and the suit are relatively well reproduced, but the face is wrong, especially the neck (in the original image) is confused with the chin (in the projected images). It is possible that the face is too small relatively to the rest of the image, compared to the FFHQ training dataset, hence the poor results of the projection.

With the same pre-processing as for the FFHQ dataset

The result below is obtained with the same pre-processing as for the FFHQ dataset, which allows to avoid the projection issues mentioned above.

Projection (without issues) as GIF

From left to right: the target image, the result obtained at the start of the projection, and the final result of the projection.

<img alt="Target image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/00001-project-real-images/image0001-target.jpg" width="250"><img alt="Projected image n°1/5" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/00001-project-real-images/image0001-step0200.jpg" width="250"><img alt="Projected image n°5/5" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/00001-project-real-images/image0001-step1000.jpg" width="250">

From left to right: the target image, the result obtained at the start of the projection, intermediate results, and the final result.

Projection results as PNG

Results: comparison with the extended projection

For the rest of the repository, the same-preprocessing as for the FFHQ dataset is used.

Shared data on Google Drive

Additional projection results are shown on the Wiki.

To make it easier to download them, they are also shared on Google Drive.

The directory structure is as follows:

stylegan2_projections/
├ aligned_images/
├ └ emmanuel-macron_01.png    # FFHQ-aligned image
├ generated_images_no_tiled/  # projections with `W(18,*)`
├ ├ emmanuel-macron_01.npy    # - latent code
├ └ emmanuel-macron_01.png    # - projected image
├ generated_images_tiled/     # projections with `W(1,*)`
├ ├ emmanuel-macron_01.npy    # - latent code
├ └ emmanuel-macron_01.png    # - projected image
├ aligned_images.tar.gz             # folder archive
├ generated_images_no_tiled.tar.gz  # folder archive
└ generated_images_tiled.tar.gz     # folder archive

Projection results

Images below allow us to compare results obtained with the original projection W(1,*) and the extended projection W(18,*).

A projected image obtained with W(18,*) is expected to be closer to the target image, at the expense of semantics.

If image fidelity is very important, W(18,*) can be run for a higher number of iterations (default is 1000 steps), but truncation might be needed for later applications.

French politicians

From top to bottom: aligned target image, projection with W(1,*), projection with W(18,*).

<img alt="Aligned target image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/aligned_images/09.jpg" width="250"><img alt="Aligned target image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/aligned_images/22.jpg" width="250"><img alt="Aligned target image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/aligned_images/32.jpg" width="250">

<img alt="W1 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_tiled/09.jpg" width="250"><img alt="W1 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_tiled/22.jpg" width="250"><img alt="W1 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_tiled/32.jpg" width="250">

<img alt="W18 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_no_tiled/09.jpg" width="250"><img alt="W18 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_no_tiled/22.jpg" width="250"><img alt="W18 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_no_tiled/32.jpg" width="250">

From top to bottom: aligned target image, projection with W(1,*), projection with W(18,*).

<img alt="Aligned target image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/aligned_images/05.jpg" width="250"><img alt="Aligned target image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/aligned_images/07.jpg" width="250"><img alt="Aligned target image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/aligned_images/38.jpg" width="250">

<img alt="W1 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_tiled/05.jpg" width="250"><img alt="W1 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_tiled/07.jpg" width="250"><img alt="W1 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_tiled/38.jpg" width="250">

<img alt="W18 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_no_tiled/05.jpg" width="250"><img alt="W18 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_no_tiled/07.jpg" width="250"><img alt="W18 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_no_tiled/38.jpg" width="250">

Art

From top to bottom: aligned target image, projection with W(1,*), projection with W(18,*).

<img alt="Aligned target image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/aligned_images/29.jpg" width="250"><img alt="Aligned target image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/aligned_images/28.jpg" width="250"><img alt="Aligned target image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/aligned_images/41.jpg" width="250">

<img alt="W1 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_tiled/29.jpg" width="250"><img alt="W1 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_tiled/28.jpg" width="250"><img alt="W1 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_tiled/41.jpg" width="250">

<img alt="W18 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_no_tiled/29.jpg" width="250"><img alt="W18 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_no_tiled/28.jpg" width="250"><img alt="W18 projected image" src="https://raw.githubusercontent.com/wiki/woctezuma/stylegan2-projecting-images/results/generated_images_no_tiled/41.jpg" width="250">

Applications

In the following, we assume that real images have been projected, so that we have access to their latent codes, of shape (1, 512) or (18, 512) depending on the projection method.

There are three main applications:

  1. morphing (linear interpolation),
  2. style transfer (crossover),
  3. expression transfer (adding a vector and a scaled difference vector).

Shared data on Google Drive

Results corresponding to each application are:

The directory structure is as follows:

stylegan2_editing/
├ expression/                   # expression transfer
| ├ no_tiled/                   # - `W(18,*)`
| | └ expression_01_age.jpg     # face n°1 ; age
| └ tiled/                      # - `W(1,*)`
|   └ expression_01_age.jpg
├ morphing/                     # morphing
| ├ no_tiled/                   # - `W(18,*)`
| | └ morphing_07_01.jpg        # face n°7 to face n°1
| └ tiled/                      # - `W(1,*)`
|   └ morphing_07_01.jpg
├ style_mixing/                 # style transfer
| ├ no_tiled/                   # - `W(18,*)`
| | └ style_mixing_42-07-10-29-41_42-07-22-39.jpg
| └ tiled/                      # - `W(1,*)`
|   └ style_mixing_42-07-10-29-41_42-07-22-39.jpg
├ video_style_mixing/           # style transfer
| ├ no_tiled/                   # - `W(18,*)`
| | └ video_style_mixing_000.000.jpg
| ├ tiled/                      # - `W(1,*)`
| | └ video_style_mixing_000.000.jpg
| ├ no_tiled_small.mp4          # with 2 reference faces
| ├ no_ tiled.mp4               # with 4 reference faces
| ├ tiled_small.mp4
| └ tiled.mp4
├ expression_transfer.tar.gz    # folder archive
├ morphing.tar.gz               # folder archive
├ style_mixing.tar.gz           # folder archive
└ video_style_mixing.tar.gz     # folder archive

1. Morphing

Morphing consists in a linear interpolation between two latent vectors (two faces).

Results are shown on the Wiki.

With the original projection W(1,*)

Morphing Morphing

Morphing Morphing

With the extended projection W(18,*)

Morphing Morphing

Morphing Morphing

2. Style transfer

Style transfer consists in a crossover of latent vectors at the layer level (cf. this piece of code).

There are 18 layers for the generator. The latent vector of the reference face is used for the first 7 layers. The latent vector of the face whose style has to be copied is used for the remaining 11 layers.

Results are shown on the Wiki.

With the original projection W(1,*)

Thanks to morphing of the faces whose style is copied, style transfer can be watched as a video.

Style Transfer

With the extended projection W(18,*)

Thanks to morphing of the faces whose style is copied, style transfer can be watched as a video.

Style Transfer

3. Expression transfer

Expression transfer consists in the addition of:

Expressions were defined, learnt, and shared on Github by a Chinese speaker:

  1. age
  2. angle_horizontal
  3. angle_pitch
  4. beauty
  5. emotion_angry
  6. emotion_disgust
  7. emotion_easy
  8. emotion_fear
  9. emotion_happy
  10. emotion_sad
  11. emotion_surprise
  12. eyes_open
  13. face_shape
  14. gender
  15. glasses
  16. height
  17. race_black
  18. race_white
  19. race_yellow
  20. smile
  21. width

Results are shown on the Wiki.

With the original projection W(1,*)

With the extended projection W(18,*)

References

<!-- Definitions -->