Awesome
ZONE: Zero-Shot Instruction-Guided Local Editing
Abstract
Recent advances in vision-language models like Stable Diffusion have shown remarkable power in creative image synthesis and editing. However, most existing text-to-image editing methods encounter two obstacles: First, the text prompt needs to be carefully crafted to achieve good results, which is not intuitive or user-friendly. Second, they are insensitive to local edits and can irreversibly affect non-edited regions, leaving obvious editing traces. To tackle these problems, we propose a Zero-shot instructiON-guided local image Editing approach, termed ZONE. We first convert the editing intent from the user-provided instruction (e.g., ``make his tie blue") into specific image editing regions through InstructPix2Pix. We then propose a Region-IoU scheme for precise image layer extraction from an off-the-shelf segment model. We further develop an edge smoother based on FFT for seamless blending between the layer and the image. Our method allows for arbitrary manipulation of a specific region with a single instruction while preserving the rest. Extensive experiments demonstrate that our ZONE achieves remarkable local editing results and user-friendliness, outperforming state-of-the-art methods.
Framework
TODO
- 2024.04.12 release inference code and test set
- 2024.04.15 update arxiv paper with camera-ready version
Dataset
We provide the test set used in our paper. We randomly select and annotate 100 samples for evaluation, including 60 real images from the Internet and 40 synthetic images. To ensure the representativeness of the evaluation, we consider the diversity of scenes and objects in the sample selection. In particular, we divide the test set into three categories: 32 images for add
, 54 for change
, and 14 for remove
actions. All these 100 images are listed in the supplementary material. The test set can be downloaded from Google Drive
Installation
Requirements
- Linux with Python ≥ 3.6
- PyTorch == 1.10 and torchvision that matches the PyTorch installation. Install them together at pytorch.org to make sure of this.
Usage
Install required packages.
conda create --name zone python=3.8
conda activate zone
cd /path/to/code
pip install -r requirements.txt
Install Segment Anything (SAM):
pip install git+https://github.com/facebookresearch/segment-anything.git
or clone the repository locally and install with
git clone git@github.com:facebookresearch/segment-anything.git
cd segment-anything; pip install -e .
Download pretrained models
Download ViT-H SAM model.
cd ckpts/
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
Other models can be directly loaded from HuggingFace.
Inference
python inference.py --instruction "make him a golden statue" --image_path "./data/dog.png"
python inference.py --instruction "wear her sunglasses" --image_path "./data/girl.png"
Accknowledgements
We thank Prompt-to-Prompt, InstructPix2Pix and Segment Anything for their great work and inspiration.
Citation
If you find this code/paper useful, please consider citing:
@article{li2023zone,
title={Zone: Zero-shot instruction-guided local editing},
author={Li, Shanglin and Zeng, Bohan and Feng, Yutang and Gao, Sicheng and Liu, Xuhui and Liu, Jiaming and Lin, Li and Tang, Xu and Hu, Yao and Liu, Jianzhuang and others},
journal={arXiv preprint arXiv:2312.16794},
year={2023}
}