Home

Awesome

<br /> <p align="center"> <h1 align="center">PhysGen: Rigid-Body Physics-Grounded <br>Image-to-Video Generation</h1> <p align="center"> ECCV, 2024 <br /> <a href="https://stevenlsw.github.io"><strong>Shaowei Liu</strong></a> · <a href="https://jason718.github.io/"><strong>Zhongzheng Ren</strong></a> · <a href="https://saurabhg.web.illinois.edu/"><strong>Saurabh Gupta*</strong></a> · <a href="https://shenlong.web.illinois.edu/"><strong>Shenlong Wang*</strong></a> · </p> <p align="center"> <img src="assets/demo.gif" alt="Demo GIF" /> </p> <p align="center"> <a href='https://arxiv.org/pdf/2409.18964'> <img src='https://img.shields.io/badge/Paper-PDF-green?style=flat&logo=arXiv&logoColor=green' alt='Paper PDF'></a> <a href='https://arxiv.org/abs/2409.18964'><img src='https://img.shields.io/badge/arXiv-2409.18964-b31b1b.svg' alt='Arxiv'></a> <a href='https://stevenlsw.github.io/physgen/' style='padding-left: 0.5rem;'> <img src='https://img.shields.io/badge/Project-Page-blue?style=flat&logo=Google%20chrome&logoColor=blue' alt='Project Page'></a> <a href='https://colab.research.google.com/drive/1imGIms3Y4RRtddA6IuxZ9bkP7N2gVVC_' style='padding-left: 0.5rem;'><img src='https://colab.research.google.com/assets/colab-badge.svg' alt='Google Colab'></a> <a href='https://youtu.be/lCc1rHePEFQ' style='padding-left: 0.5rem;'> <img src='https://img.shields.io/badge/Youtube-Video-red?style=flat&logo=youtube&logoColor=red' alt='Youtube Video'></a> </p> </p> <br />

This repository contains the pytorch implementation for the paper PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation, ECCV 2024. In this paper, we present a novel training-free image-to-video generation pipeline integrates physical simulation and generative video diffusion prior.

Overview

overview

📄 Table of Contents

Installation

Colab Notebook

Run our Colab notebook for quick start!

Quick Demo

Perception

InputSegmentationNormalAlbedoShadingInpainting
<img src="data/pig_ball/original.png" alt="input" width="100"/><img src="data/pig_ball/vis.png" alt="segmentation" width="100"/><img src="data/pig_ball/intermediate/normal_vis.png" alt="normal" width="100"/><img src="data/pig_ball/intermediate/albedo_vis.png" alt="albedo" width="100"/><img src="data/pig_ball/intermediate/shading_vis.png" alt="shading" width="100"/><img src="data/pig_ball/inpaint.png" alt="inpainting" width="100"/>

Simulation

Rendering

Relighting

Video Diffusion Rendering

All-in-One command

We integrate the simulation, relighting and video diffusion rendering in one script. Please follow the Video Diffusion Rendering to download the SEINE model first.

bash scripts/run_demo.sh ${name}

Evaluation

We compare ours against open-sourced img-to-video models DynamiCrafter, I2VGen-XL, SEINE and collected reference videos GT in Sec. <font color="red">4.3</font>.

Custom Image Video Generation

Citation

If you find our work useful in your research, please cite:

@inproceedings{liu2024physgen,
  title={PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation},
  author={Liu, Shaowei and Ren, Zhongzheng and Gupta, Saurabh and Wang, Shenlong},
  booktitle={European Conference on Computer Vision ECCV},
  year={2024}
}

Acknowledgement