Home

Awesome

ShadowGP

This repository provides the official codes for our paper: Unsupervised Portrait Shadow Removal via Generative Priors (ACM MM 2021).

Unsupervised Portrait Shadow Removal via Generative Priors <br> Yingqing He*, Yazhou Xing*, Tianjia Zhang, Qifeng Chen (* indicates joint first authors)<br> HKUST <br>

<!-- [[Paper](https://arxiv.org/abs/2108.03466)] --> <!-- [[Project Page (Coming soon)](TBA)] [[Technical Video (Coming soon)](TBA)] -->

In this repository, we propose an unsupervised method for portrait shadow removal, named as ShadowGP. ShadowGP can recover a shadow-free portrait image via single image optimization, without a large paired training dataset, which is expensive to collect and time-consuming to train. Besides, our method can also be extended to facial tattoo removal and watermark removal tasks.

<!-- **Figure:** *Our results* --> <!-- <br /> -->

ShadowGP can decompose the single input shadowed portrait image into 3 parts: a full-shadow portrait, a shadow-free portrait and a shadow mask. Blending the three parts together can reconstruct the input shadowed portrait. The decomposed shadow-free portrait is the target output.

<!-- **Figure:** *Our unsupervised method takes a single shadow portrait as input and can decompose it into a shadow-free portrait image, a full-shadow portrait image, and a shadow mask* --> <br />

Install Environment

To install and activate the environment, run the following commands:

conda create -n shadowgp python=3.7
conda activate shadowgp
pip3 install torch==1.3.1+cu100 torchvision==0.4.2+cu100 -f https://download.pytorch.org/whl/torch_stable.html
pip3 install opencv-python tqdm scikit-image==0.15 Pillow==7.1.0 Ipython ninja

Our code has the same system requirements as StyleGAN2-PyTorch.

<br />

Download Checkpoints

Download the pretrained StyleGAN 550000.pt and the pretrained face segmentation model face-seg-BiSeNet-79999_iter.pth from the google drive, and put them in checkpoint/.

cd Shadow-Removal-via-Generative-Priors
mkdir checkpoint
mv ${YOUR_PATH}/550000.pt checkpoint/
mv ${YOUR_PATH}/face-seg-BiSeNet-79999_iter.pth checkpoint/

Optional: If the lpips checkpoint cannot be downloaded automatically, you can also manually download the weights.tar.gz to the project directory and run tar -zxvf weights.tar.gz. Then you will see the checkpoint in lpips/weights/v0.1/vgg.pth.

Run

bash run.sh

Then it will create a results folder including all the intermidiate results and the final output results in results/res. <br />

Acknowledgement

Our code is built on StyleGAN2-PyTorch.

<br />

Citation

@inproceedings{he21unsupervised,
  title     = {Unsupervised Portrait Shadow Removal via Generative Priors},
  author    = {He, Yingqing and Xing, Yazhou and Zhang, Tianjia and Chen, Qifeng},
  booktitle = {ACM International Conference on Multimedia (ACM MM)},
  year      = {2021}
}