Awesome
<h1 align='center'>HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models</h1> <div align='center'> <a href='https://github.com/songkey' target='_blank'>Shengkai Zhang</a>, <a href='https://github.com/RhythmJnh' target='_blank'>Nianhong Jiao</a>, <a href='https://github.com/Shelton0215' target='_blank'>Tian Li</a>, <a href='https://github.com/chaojie12131243' target='_blank'>Chaojie Yang</a>, <a href='https://github.com/xchgit' target='_blank'>Chenhui Xue</a><sup>*</sup>, <a href='https://github.com/boya34' target='_blank'>Boya Niu</a><sup>*</sup>, <a href='https://github.com/HelloVision/HelloMeme' target='_blank'>Jun Gao</a> </div> <div align='center'> HelloVision | HelloGroup Inc. </div> <div align='center'> <small><sup>*</sup> Intern</small> </div> <br> <div align='center'> <a href='https://songkey.github.io/hellomeme/'><img src='https://img.shields.io/badge/Project-HomePage-Green'></a> <a href='https://arxiv.org/pdf/2410.22901'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> <a href='https://huggingface.co/songkey'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Model-yellow'></a> <a href='https://github.com/HelloVision/HelloMeme'><img src='https://img.shields.io/badge/GitHub-Code-blue'></a> </div>🔆 New Features/Updates
-
✅
11/14/2024Added theHMControlNet2module, which uses thePD-FGCmotion module to extract facial expression information (drive_exp2); restructured the ComfyUI interface; and optimized VRAM usage.
-
✅
11/12/2024Added a newly fine-tuned version ofAnimatediffwith a patch size of 12, which uses less VRAM (Tested on 2080Ti). -
✅
11/11/2024Optimized VRAM usage and addedHMVideoSimplePipeline(workflows/hellomeme_video_simple_workflow.json), which doesn’t use Animatediff and can run on machines with less than 12G VRAM. -
✅
11/6/2024The face proportion in the reference image significantly affects the generation quality. We have encapsulated the recommended image cropping method used during training into aCropReferenceImageNode. Refer to the workflows in theComfyUI_HelloMeme/workflows directory:hellomeme_video_cropref_workflow.jsonandhellomeme_image_cropref_workflow.json.
Introduction
This repository is the official implementation of the HelloMeme ComfyUI interface, featuring both image and video generation functionalities. Example workflow files can be found in the ComfyUI_HelloMeme/workflows directory. Test images and videos are saved in the ComfyUI_HelloMeme/examples directory. Below are screenshots of the interfaces for image and video generation.
[!Note] Custom models should be placed in the directories listed below.
Checkpoints under:
ComfyUI/models/checkpointsLoras under:
ComfyUI/models/checkpoints