Home

Awesome

Pluralistic Image Completion

ArXiv | Project Page | Online Demo | Video(demo) <br>

This repository implements the training, testing and editing tools for "Pluralistic Image Completion" by Chuanxia Zheng, Tat-Jen Cham and Jianfei Cai at NTU. Given one masked image, the proposed Pluralistic model is able to generate multiple and diverse plausible results with various structure, color and texture.

Editing example

<img src='images/free_form.gif' align="center">

Example results

<table> <tr> <td><img src='images/mask_celeba_185755.jpg'></td> <td><img src='images/celeba185755.gif'></td> <td><img src='images/mask_celeba_184919.jpg'></td> <td><img src='images/celeba184919.gif'></td> </tr> <tr> <td><img src='images/mask_paris_085.png'></td> <td><img src='images/paris85.gif'></td> <td><img src='images/mask_Places_00030002.jpg'></td> <td><img src='images/place30002.gif'></td> </tr> </table>

Example completion results of our method on images of face (CelebA), building (Paris), and natural scenes (Places2) with center masks (masks shown in gray). For each group, the masked input image is shown left, followed by sampled results from our model without any post-processing. The results are diverse and plusible.

More results on project page

Getting started

Installation

This code was tested with Pytoch 0.4.0, CUDA 9.0, Python 3.6 and Ubuntu 16.04

The GUI program was further verified compatible with PyTorch 1.7.0, CUDA 11.2, Python 3.6 and Ubuntu 20.04.

pip install visdom dominate
git clone https://github.com/lyndonzheng/Pluralistic
cd Pluralistic

Datasets

Training

python train.py --name celeba_random --img_file your_image_path

Testing

python test.py  --name celeba_random --img_file your_image_path

Pretrained Models

Download the pre-trained models using the following links and put them undercheckpoints/ directory.

Our main novelty of this project is the multiple and diverse plausible results for one given masked image. The center_mask models are trained with images of resolution 256*256 with center holes 128x128, which have large diversity for the large missing information. The random_mask models are trained with random regular and irregular holes, which have different diversity for different mask sizes and image backgrounds.

GUI

<img src='images/free_form_v2.gif' align="center">
pip install PyQt5
pip install face_alignment==1.3.5  

Basic usage is:

python -m visdom.server
python ui_main.py

The buttons in GUI:

The steps are as follows:

1. Select a model from 'options'
2. Click the 'random', 'load' or 'capture' button to get an input image.
3. [Optional] If the image is loaded by 'load' or 'capture', and the program is running with human face model, click the 'crop face' button to automatically detect and align face. Do NOT apply 'crop face' if image is loaded by 'random'.
4. If you choose a random model, click the 'draw/clear' button to input free_form mask.
5. If you choose a center model, the center mask has been given.
6. click 'fill' button to get multiple results.
7. click 'save' button to save the results.

Editing Example Results

<table> <tr> <td><img src='images/removing/original_celeba189756.jpg'></td> <td><img src='images/removing/original_celeba199782.jpg'></td> <td><img src='images/removing/original_paris085.png'></td> <td><img src='images/removing/original_place00013547.jpg'></td> </tr> <tr> <td><img src='images/removing/mask_celeba189756.jpg'></td> <td><img src='images/removing/mask_celeba199782.jpg'></td> <td><img src='images/removing/mask_paris085.png'></td> <td><img src='images/removing/mask_place00013547.jpg'></td> </tr> <tr> <td><img src='images/removing/result_celeba189756.jpg'></td> <td><img src='images/removing/result_celeba199782.jpg'></td> <td><img src='images/removing/result_paris085.png'></td> <td><img src='images/removing/result_place00013547.jpg'></td> </tr> </table> <table> <tr> <td><img src='images/face_playing/mask_celeba184054.jpg'></td> <td><img src='images/face_playing/result_celeba184054.gif'></td> <td><img src='images/face_playing/mask_celeba182927.jpg'></td> <td><img src='images/face_playing/result_celeba182927.gif'></td> </tr> <tr> <td><img src='images/face_playing/mask_celeba192793.jpg'></td> <td><img src='images/face_playing/result_celeba192793.gif'></td> <td><img src='images/face_playing/mask_celeba197462.jpg'></td> <td><img src='images/face_playing/result_celeba197462.gif'></td> </tr> <tr> <td><img src='images/face_playing/mask_celeba198496.jpg'></td> <td><img src='images/face_playing/result_celeba198496.jpg'></td> <td><img src='images/face_playing/mask_celeba190952.jpg'></td> <td><img src='images/face_playing/result_celeba190952.jpg'></td> </tr> </table>

Next

License

<br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc/4.0/">Creative Commons Attribution-NonCommercial 4.0 International License</a>.

This software is for educational and academic research purpose only. If you wish to obtain a commercial royalty bearing license to this software, please contact us at chuanxia001@e.ntu.edu.sg.

Citation

If you use this code for your research, please cite our paper.

@inproceedings{zheng2019pluralistic,
  title={Pluralistic Image Completion},
  author={Zheng, Chuanxia and Cham, Tat-Jen and Cai, Jianfei},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={1438--1447},
  year={2019}
}

@article{zheng2021pluralistic,
  title={Pluralistic Free-From Image Completion},
  author={Zheng, Chuanxia and Cham, Tat-Jen and Cai, Jianfei},
  journal={International Journal of Computer Vision},
  pages={1--20},
  year={2021},
  publisher={Springer}
}