Awesome
<h1 align='center'>HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models</h1> <div align='center'> <a href='https://github.com/songkey' target='_blank'>Shengkai Zhang</a>, <a href='https://github.com/RhythmJnh' target='_blank'>Nianhong Jiao</a>, <a href='https://github.com/Shelton0215' target='_blank'>Tian Li</a>, <a href='https://github.com/chaojie12131243' target='_blank'>Chaojie Yang</a>, <a href='https://github.com/xchgit' target='_blank'>Chenhui Xue</a><sup>*</sup>, <a href='https://github.com/boya34' target='_blank'>Boya Niu</a><sup>*</sup>, <a href='https://github.com/HelloVision/HelloMeme' target='_blank'>Jun Gao</a> </div> <div align='center'> HelloVision | HelloGroup Inc. </div> <div align='center'> <small><sup>*</sup> Intern</small> </div> <br> <div align='center'> <a href='https://songkey.github.io/hellomeme/'><img src='https://img.shields.io/badge/Project-HomePage-Green'></a> <a href='https://arxiv.org/pdf/2410.22901'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> <a href='https://huggingface.co/songkey'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Model-yellow'></a> <a href='https://github.com/HelloVision/HelloMeme'><img src='https://img.shields.io/badge/GitHub-Code-blue'></a> <a href='https://www.modelscope.cn/studios/songkey/HelloMeme'><img src='https://img.shields.io/badge/modelscpe-Demo-red'></a> </div>π New Features/Updates
-
β
12/17/2024
Support modelscope (Modelscope Demo). -
β
12/08/2024
Added HelloMemeV2 (select "v2" in the version option of the LoadHelloMemeImage/Video Node). Its features include: a. Improved expression consistency between the generated video and the driving video. b. Better compatibility with third-party checkpoints (we will continuously collect compatible free third-party checkpoints and share them on this page. If you'd like to recommend one, please post it on the issue page or email us at songkey@pku.edu.cn). c. Reduced VRAM usage. -
β
11/29/2024
a.Optimize the algorithm; b.Add VAE selection functionality; c.Introduce a super-resolution feature. YouTube Demo -
β
11/14/2024
Added theHMControlNet2
module, which uses thePD-FGC
motion module to extract facial expression information (drive_exp2
); restructured the ComfyUI interface; and optimized VRAM usage. YouTube Demo -
β
11/12/2024
Added a newly fine-tuned version ofAnimatediff
with a patch size of 12, which uses less VRAM (Tested on 2080Ti). -
β
11/11/2024
Optimized VRAM usage and addedHMVideoSimplePipeline
(workflows/hellomeme_video_simple_workflow.json
), which doesnβt use Animatediff and can run on machines with less than 12G VRAM. -
β
11/6/2024
The face proportion in the reference image significantly affects the generation quality. We have encapsulated the recommended image cropping method used during training into aCropReferenceImage
Node. Refer to the workflows in theComfyUI_HelloMeme/workflows directory
:hellomeme_video_cropref_workflow.json
andhellomeme_image_cropref_workflow.json
.
Introduction
This repository is the official implementation of the HelloMeme
ComfyUI interface, featuring both image and video generation functionalities. Example workflow files can be found in the ComfyUI_HelloMeme/workflows
directory. Test images and videos are saved in the ComfyUI_HelloMeme/examples
directory. Below are screenshots of the interfaces for image and video generation.
[!Note] Custom models should be placed in the directories listed below.
Checkpoints under:
ComfyUI/models/checkpoints
Loras under:
ComfyUI/models/checkpoints
Recommended Third-party Checkpoints/Loras
Name | Checkpoints/Loras | Recommenders |
---|---|---|
realisticVisionV60B1_v51VAE | Checkpoints | |
disneyPixarCartoon_v10 | Checkpoints |
Workflows
workflow file | Video Generation | Image Generation | HMControlNet | HMControlNet2 | Super-Resolution |
---|---|---|---|---|---|
hellomeme_image_workflow.json | β | β | |||
hellomeme_video_workflow.json | β | β | |||
hellomeme_image_sr_workflow.json | β | β | β | ||
hellomeme_video_sr_workflow.json | β | β | β | ||
hellomeme_image_v2_workflow.json | β | β | |||
hellomeme_video_v2_workflow.json | β | β | |||
hellomeme_image_v2_sr_workflow.json | β | β | β | ||
hellomeme_video_v2_sr_workflow.json | β | β | β |