Awesome
ComfyUI-MochiEdit
ComfyUI nodes to edit videos using Genmo Mochi
https://github.com/user-attachments/assets/41830ff3-6ac6-4b5a-be35-4429c571aa97
Installation
These nodes are built to work with the ComfyUI-MochiWrapper nodes and soon will work with native ComfyUI Mochi too. For now please follow the installation for the wrapper.
Then git clone this repo into your ComfyUI/custom_nodes/
directory or use the ComfyUI Manager to install (when this repo is added there).
There are no additional requirements.
https://github.com/user-attachments/assets/88a9c4d4-a6d2-4d68-9c07-7fcba32ce84a
How to Use
There is an example workflow in the example_workflows
directory.
First, the input video is inverted into noise and then this noise is used to resample the video with the target prompt. A similar strategy as RF-Inversion is used.
Unsampling Nodes
<img width="993" alt="unsampling_nodes" src="https://github.com/user-attachments/assets/abd63fd8-0681-419f-a209-dc7dc769e8cf">Mochi Unsampler
This node creates a sampler that can convert the video into noise.
gamma
: the amount to do noise correction. Leave this to 0 as it does not work well with Mochi.seed
: if performing noise correction the seed to use for the random noise
Mochi Prepare Sigmas
This node makes a small change to the sigmas that the Mochi Sigma Schedule node from the wrapper produces.
SamplerCustom (MochiWrapper)
This is the classic KSampler or SamplerCustom from ComfyUI but for the MochiWrapper.
positive
andnegative
should be blank promptscfg
: should always be 1.0 for unsamplingadd_noise
: should always be False for unsamplingseed
: there is no reason to change the seedsigmas
: must be prepared then flipped first
Sampling Nodes
<img width="803" alt="sampling_nodes" src="https://github.com/user-attachments/assets/3f9606ef-0a3b-4000-8154-c02c80b8402a">Mochi Resampler
This node creates a sampler that can convert the noise into a video.
latents
: the latents of the original videoeta
: the strength that the generation should align with the original video- higher values lead the generation closer to the original
start_step
: the starting step to where the original video should guide the generation- a lower value (e.g. 0) will have much closer following but not allow for additional objects like a hat to be placed
- a higher value (e.g. 6) will allow for new objects like a hat to be placed, but may not follow the original video. Higher values can also lead to bad results (blurs)
end_step
the step to stop guiding the generation closer to the original video- a lower value will lead to more differences in the video output
eta_trend
: whether the eta (strength of guidance) should stay constant, increase, or decrease as steps progress.linear_decrease
is the recommended setting for most changes.
SamplerCustom (MochiWrapper)
This is the classic KSampler or SamplerCustom from ComfyUI but for the MochiWrapper.
positive
andnegative
can be anything you like.positive
shoud be the target prompt.cfg
: can have any cfg that would work with normal Mochi (e.g. 4.50)latents
: should be the latent from unsamplingsigmas
: must be prepared but NOT flippedseed
: the seed has no effect
Acknowledgements
@article{rout2024rfinversion,
title={Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations},
author={Litu Rout and Yujia Chen and Nataniel Ruiz and Constantine Caramanis and Sanjay Shakkottai and Wen-Sheng Chu},
journal={arXiv preprint arXiv:2410.10792},
year={2024}
}
https://github.com/user-attachments/assets/d1d8e73a-680d-4671-b5f0-b2efd7ac05f2