Home

Awesome

PAL4Inpaint

[Project Page] | Paper | Bibtex

<img src="https://github.com/owenzlz/PAL4Inpaint/blob/main/images/teaser.png" style="width:800px;">

Perceptual Artifacts Localization for Inpainting
European Conference on Computer Vision (ECCV), 2022, Oral Presentation
Lingzhi Zhang, Yuqian Zhou, Connelly Barnes, Sohrab Amirghodsi, Zhe Lin, Eli Shechtman, Jianbo Shi
University of Pennsylvania, Adobe Research/ART

Note: Due to commercial reasons, we release only the inference code and a subset of the datasets.

Prerequisites

Setup

git clone https://github.com/owenzlz/PA4Inpaint
pip install torch torchvision

Datasets

You can download our labeled datasets using the following command.

<img src="https://github.com/owenzlz/PA4Inpaint/blob/main/images/user_labels.png" style="width:600px;">
bash download_datasets.sh

After downloading, the dataset is structured as follows:

- [perceptual artifacts dataset root]
    |- trainset_subset
        |- images
        |- masks
        |- labels
    |- valset
        |- images
        |- masks
        |- labels
    |- testset
        |- images
        |- masks
        |- labels

Note that labels and masks have pixel values 0 or 1. To visualize them, you can multiple the images by 255. We provide a simple script to visualize a few sampled data.

python visualize_labels.py

Checkpoints

bash download_checkpoints.sh

Perceptual Artifacts Segmentation

Run the following command to predict the perceptual artifacts localization on the inpainted images.

python pal4inpaint.py \
       --img_file ./testimages/images/a_alcove_00004858.png \
       --output_seg_file ./testimages/seg/a_alcove_00004858.png \
       --output_vis_file ./testimages/vis/a_alcove_00004858.png
python pal4inpaint.py \
       --img_dir ./demo/images \
       --output_seg_dir ./demo/results/seg \
       --output_vis_dir ./demo/results/vis
<img src="https://github.com/owenzlz/PA4Inpaint/blob/main/demo/artifacts_seg.gif" style="width:900px;">

Perceptual Artifacts Ratio (PAR) [TODO]

python par.py --img_file ./demo/images/xxx.jpg
python par.py --img_dir ./demo/images

Iterative Fill [TODO]

python iterative_fill.py
<img src="https://github.com/owenzlz/PA4Inpaint/blob/main/demo/iterative_fill.png" style="width:900px;">

The blue contour in the first image indicates the original hole, and the pink contours in each images indicate the localized perceptual artifacts region at each iterative fill.

Citation

If you use this code for your research, please cite our paper:

@article{zhang2022perceptual,
  title={Perceptual Artifacts Localization for Inpainting},
  author={Zhang, Lingzhi and Zhou, Yuqian and Barnes, Connelly and Amirghodsi, Sohrab and Lin, Zhe and Shechtman, Eli and Shi, Jianbo},
  journal={arXiv preprint arXiv:2208.03357},
  year={2022}
}