Awesome

https://github.com/wenhao728/awesome-diffusion-v2v/assets/65353366/fd42e40f-265d-4d72-8dc1-bf74d00fe87b

📌 Table of Contents

• Introduction
• Network and Training Paradigm
  • Temporal Adaption
  • Structure Conditioning
  • Training Modification
• Attention Feature Injection
  • Inversion-Based Feature Injection
  • Motion-Based Feature Injection
• Diffusion Latents Manipulation
  • Latent Initialization
  • Latent Transition
• Canonical Representation
• Novel Conditioning
  • Point-Based Editing
  • Pose-Guided Human Action Editing

Introduction

The diffusion process defines a Markov chain that progressively adds random noise to data and learns to reverse this process to generate desired data samples from noise. Deep neural networks facilitate the transitions between latent states.

• In Network and Training Paradigm, we explore neural network modifications to adapt diffusion models for video editing tasks.
• Attention Feature Injection introduces a training-free inference-time technique.
• Diffusion Latents Manipulation methods treat the neural network as a black box, focusing on the diffusion process itself.
• We then list Canonical Representation methods for efficient video representation.
• Finally, Novel Conditioning methods cover innovative conditioning techniques for video editing tasks.

Network and Training Paradigm

Temporal Adaption

Structure Conditioning

Training Modification

Attention Feature Injection

Inversion-Based Feature Injection

Motion-Based Feature Injection

Diffusion Latents Manipulation

Latent Initialization

Latent Transition

Canonical Representation

Novel Conditioning

Point-Based Editing

Pose-Guided Human Action Editing

📈 V2VBench

Leaderboard

V2VBench is a comprehensive benchmark designed to evaluate video editing methods. It consists of:

• 50 standardized videos across 5 categories, and
• 3 editing prompts per video, encompassing 4 editing tasks: Huggingface Datasets
• 8 evaluation metrics to assess the quality of edited videos: Evaluation Metrics

For detailed information, please refer to the accompanying paper.

🍻 Citation

If you find this repository helpful, please consider citing our paper:

@article{sun2024v2vsurvey,
    author = {Wenhao Sun and Rong-Cheng Tu and Jingyi Liao and Dacheng Tao},
    title = {Diffusion Model-Based Video Editing: A Survey},
    journal = {CoRR},
    volume = {abs/2407.07111},
    year = {2024}
}