Home

Awesome

📷 EasyPhoto | Your Smart AI Photo Generator.

🦜 EasyPhoto is a Webui UI plugin for generating AI portraits that can be used to train digital doppelgangers relevant to you.

🦜 🦜 Welcome!

Hugging Face Spaces

English | 简体中文

Table of Contents

Introduction

EasyPhoto is a Webui UI plugin for generating AI portraits that can be used to train digital doppelgangers relevant to you. Training is recommended to be done with 5 to 20 portrait images, preferably half-body photos, and do not wear glasses (It doesn't matter if the characters in a few pictures wear glasses). After the training is done, we can generate it in the Inference section. We support using preset template images or uploading your own images for Inference.

Please read our Contributor Covenant covenant | 简体中文.

If you encounter any problems in the training, please refer to the VQA.

We now support quick pull-ups from different platforms, refer to Quick Start.

Now you can experience EasyPhoto demo quickly on ModelScope, demo.

What's New:

These are our generated results: results_1

Video Part:

Example123
-<img src="http://pai-vision-data-hz.oss-accelerate.aliyuncs.com/easyphoto/data/video/text2video/51s3.gif" width="400"><img src="http://pai-vision-data-hz.oss-accelerate.aliyuncs.com/easyphoto/data/video/v2videos/ring_3644.gif" width="400"><img src="http://pai-vision-data-hz.oss-accelerate.aliyuncs.com/easyphoto/data/video/img2video_2imgs/29s3.gif" width="400">

Photo Part: results_2 results_3

Our UI interface is as follows: train part: train_ui inference part: infer_ui

TODO List

Quick Start

1. Cloud usage: AliyunDSW/AutoDL/lanrui-ai/Docker

a. From AliyunDSW

DSW has free GPU time, which can be applied once by a user and is valid for 3 months after applying.

Aliyun provide free GPU time in Freetier, get it and use in Aliyun PAI-DSW to start EasyPhoto within 3min!

DSW Notebook

b. From AutoDL/lanrui-ai

lanrui-ai

The official full-plugin version of lanrui-ai comes with EasyPhoto built-in. They promise bi-weekly testing and updates. Personally tested and found to be effective, it can be launched within 5 minutes. Thanks to their support and contributions to the community.

AutoDL

If you are using Lanrui-ai/AutoDL, you can quickly pull up the Stable DIffusion webui using the mirror we provide.

You can select the desired mirror by filling in the following information in Community Mirrors, or using offical Image provide by lanrui-ai.

aigc-apps/sd-webui-EasyPhoto/sd-webui-EasyPhoto

c. From docker

If you are using docker, please make sure that the graphics card driver and CUDA environment have been installed correctly in your machine.

Then execute the following commands in this way:

# pull image
docker pull registry.cn-beijing.aliyuncs.com/mybigpai/sd-webui-easyphoto:0.0.3

# enter image
docker run -it -p 7860:7860 --network host --gpus all registry.cn-beijing.aliyuncs.com/mybigpai/sd-webui-easyphoto:0.0.3

# launch webui
python3 launch.py --port 7860

The docker updates may be slightly slower than the github repository of sd-webui-EasyPhoto, so you can go to extensions/sd-webui-EasyPhoto and do a git pull first.

cd extensions/sd-webui-EasyPhoto/
git pull
cd /workspace

2. Local install: Environment Check/Downloading/Installation

a. Environment Check

We have verified EasyPhoto execution on the following environment: If you meet problem with WebUI auto killed by OOM, please refer to ISSUE21, and setting some num_threads to 0 and report other fix to us, thanks.

The detailed of Windows 10:

The detailed of Linux:

We need about 60GB available on disk (for saving weights and datasets process), please check!

b. Relevant Repositories & Weights Downloading

i. Controlnet

We need to use Controlnet for inference. The related repo is Mikubill/sd-webui-controlnet. You need install this repo before using EasyPhoto.

In addition, we need at least three Controlnets for inference. So you need to set the Multi ControlNet: Max models amount (requires restart) in Setting. controlnet_num

ii. Other Dependencies.

We are mutually compatible with the existing stable-diffusion-webui environment, and the relevant repositories are installed when starting stable-diffusion-webui.

The weights we need will be downloaded automatically when you start training first time.

c. Plug-in Installation

We now support installing EasyPhoto from git. The url of our Repository is https://github.com/aigc-apps/sd-webui-EasyPhoto.

We will support installing EasyPhoto from Available in the future.

install

How to use

1. Model Training

The EasyPhoto training interface is as follows:

After clicking Upload Photos, we can start uploading images. It is best to upload 5 to 20 images here, including different angles and lighting conditions. It is best to have some images that do not include glasses. If they are all glasses, the generated results may easily generate glasses. train_1

Then we click on Start Training below, and at this point, we need to fill in the User ID above, such as the user's name, to start training. train_2

After the model starts training, the webui will automatically refresh the training log. If there is no refresh, click Refresh Log button. train_3

If you want to set parameters, the parsing of each parameter is as follows:

Parameter NameMeaning
ResolutionThe size of the image fed into the network during training, with a default value of 512
Validation & save stepsThe number of steps between validating the image and saving intermediate weights, with a default value of 100, representing verifying the image every 100 steps and saving the weights
Max train stepsMaximum number of training steps, default value is 800
Max steps per photosThe maximum number of training sessions per image, default to 200
Train batch sizeThe batch size of the training, with a default value of 1
Gradient accumulation stepsWhether to perform gradient accumulation. The default value is 4. Combined with the train batch size, each step is equivalent to feeding four images
Dataloader num workersThe number of jobs loaded with data, which does not take effect under Windows because an error will be reported if set, but is set normally on Linux
Learning rateTrain Lora's learning rate, default to 1e-4
Rank LoraThe feature length of the weight, default to 128
Network alphaThe regularization parameter for Lora training, usually half of the rank, defaults to 64

2. Inference

a. single people

single_people

b. multi people

single_people single_people

Algorithm Detailed

1. Architectural Overview

overview

In the field of AI portraits, we expect model-generated images to be realistic and resemble the user, and traditional approaches introduce unrealistic lighting (such as face fusion or roop). To address this unrealism, we introduce the image-to-image capability of the stable diffusion model. Generating a perfect personal portrait takes into account the desired generation scenario and the user's digital doppelgänger. We use a pre-prepared template as the desired generation scene and an online trained face LoRA model as the user's digital doppelganger, which is a popular stable diffusion fine-tuning model. We use a small number of user images to train a stable digital doppelgänger of the user, and generate a personal portrait image based on the face LoRA model and the expected generative scene during inference.

2. Training Detailed

overview

First, we perform face detection on the input user image, and after determining the face location, we intercept the input image according to a certain ratio. Then, we use the saliency detection model and the skin beautification model to obtain a clean face training image, which basically consists of only faces. Then, we label each image with a fixed label. There is no need to use a labeler here, and the results are good. Finally, we fine-tune the stabilizing diffusion model to get the user's digital doppelganger.

During training, we utilize the template image for verification in real time, and at the end of training, we calculate the face id gap between the verification image and the user's image to achieve Lora fusion, which ensures that our Lora is a perfect digital doppelganger of the user.

In addition, we will choose the image that is most similar to the user in the validation as the face_id image, which will be used in Inference.

3. Inference Detailed

a. First Diffusion:

First, we will perform face detection on our incoming template image to determine the mask that needs to be inpainted for stable diffusion. then we will use the template image to perform face fusion with the optimal user image. After the face fusion is completed, we use the above mask to inpaint (fusion_image) with the face fused image. In addition, we will affix the optimal face_id image obtained during training to the template image by affine transformation (replaced_image). Then we will apply Controlnets on it, we use canny with color to extract features for fusion_image and openpose for replaced_image to ensure the similarity and stability of the images. Then we will use Stable Diffusion combined with the user's digital split for generation.

b. Second Diffusion:

After getting the result of First Diffusion, we will fuse the result with the optimal user image for face fusion, and then we will use Stable Diffusion again with the user's digital doppelganger for generation. The second generation will use higher resolution.

Special thanks

Special thanks to DevelopmentZheng, qiuyanxin, rainlee, jhuang1207, bubbliiiing, wuziheng, yjjinjie, hkunzhe, yunkchen for their code contributions (in no particular order).

Reference

Related Project

We've also listed some great open source projects as well as any extensions you might be interested in:

License

This project is licensed under the Apache License (Version 2.0).

ContactUS

  1. Use Dingding to search group-2 54095000124 or Scan to join
  2. Since the WeChat group is full, you need to scan the image on the right to add this student as a friend first, and then join the WeChat group.
<figure> <img src="images/ding_erweima.jpg" width=300/> <img src="images/wechat.jpg" width=300/> </figure>

Contributors ✨

Thanks goes to these wonderful people :

<table> <tr> <td> <a href="https://github.com/aigc-apps/sd-webui-EasyPhoto/graphs/contributors"> <img src="https://contrib.rocks/image?repo=aigc-apps/sd-webui-EasyPhoto" /> </a> </td> </tr> </table>

This project follows the all-contributors specification. Contributions of any kind are welcome!

<p align="right"><a href="#top">Back to top</a></p>