Awesome
PAL4Inpaint
[Project Page] | Paper | Bibtex
<img src="https://github.com/owenzlz/PAL4Inpaint/blob/main/images/teaser.png" style="width:800px;">Perceptual Artifacts Localization for Inpainting
European Conference on Computer Vision (ECCV), 2022, Oral Presentation
Lingzhi Zhang, Yuqian Zhou, Connelly Barnes, Sohrab Amirghodsi, Zhe Lin, Eli Shechtman, Jianbo Shi
University of Pennsylvania, Adobe Research/ART
Note: Due to commercial reasons, we release only the inference code and a subset of the datasets.
Prerequisites
- Linux
- Python 3
- NVIDIA GPU + CUDA CuDNN
Setup
- Clone this repo:
git clone https://github.com/owenzlz/PA4Inpaint
- Install dependencies (Since we use torchscript, we can bypass the mmsegmentation packages.):
pip install torch torchvision
Datasets
You can download our labeled datasets using the following command.
<img src="https://github.com/owenzlz/PA4Inpaint/blob/main/images/user_labels.png" style="width:600px;">- Download our datasets
bash download_datasets.sh
After downloading, the dataset is structured as follows:
- [perceptual artifacts dataset root]
|- trainset_subset
|- images
|- masks
|- labels
|- valset
|- images
|- masks
|- labels
|- testset
|- images
|- masks
|- labels
Note that labels and masks have pixel values 0 or 1. To visualize them, you can multiple the images by 255. We provide a simple script to visualize a few sampled data.
python visualize_labels.py
Checkpoints
- Download checkpoints in the 'torchscript' format:
bash download_checkpoints.sh
Perceptual Artifacts Segmentation
Run the following command to predict the perceptual artifacts localization on the inpainted images.
- Inference on a single image:
python pal4inpaint.py \
--img_file ./testimages/images/a_alcove_00004858.png \
--output_seg_file ./testimages/seg/a_alcove_00004858.png \
--output_vis_file ./testimages/vis/a_alcove_00004858.png
- Inference on a batch of images:
python pal4inpaint.py \
--img_dir ./demo/images \
--output_seg_dir ./demo/results/seg \
--output_vis_dir ./demo/results/vis
<img src="https://github.com/owenzlz/PA4Inpaint/blob/main/demo/artifacts_seg.gif" style="width:900px;">
Perceptual Artifacts Ratio (PAR) [TODO]
- Compute PAR metric on a single image:
python par.py --img_file ./demo/images/xxx.jpg
- Compute PAR metric on a batch of images:
python par.py --img_dir ./demo/images
Iterative Fill [TODO]
- Localize perceptual artifacts region and run inpainting on it iteratively.
python iterative_fill.py
<img src="https://github.com/owenzlz/PA4Inpaint/blob/main/demo/iterative_fill.png" style="width:900px;">
The blue contour in the first image indicates the original hole, and the pink contours in each images indicate the localized perceptual artifacts region at each iterative fill.
Citation
If you use this code for your research, please cite our paper:
@article{zhang2022perceptual,
title={Perceptual Artifacts Localization for Inpainting},
author={Zhang, Lingzhi and Zhou, Yuqian and Barnes, Connelly and Amirghodsi, Sohrab and Lin, Zhe and Shechtman, Eli and Shi, Jianbo},
journal={arXiv preprint arXiv:2208.03357},
year={2022}
}