Awesome
StreamV2V
Looking Backward: Streaming Video-to-Video Translation with Feature Banks <br/> Feng Liang, Akio Kodaira, Chenfeng Xu, Masayoshi Tomizuka, Kurt Keutzer, Diana Marculescu <br/>
Highlight
Our StreamV2V could perform real-time video-2-video translation on one RTX 4090 GPU. Check the video and try it by yourself!
For functionality, our StreamV2V supports face swap (e.g., to Elon Musk or Will Smith) and video stylization (e.g., to Claymation or doodle art). Check the video and reproduce the results!
Although StreamV2V is designed for the vid2vid task, it could seamlessly integrate with the txt2img application. Compared with per-image StreamDiffusion, StreamV2V continuously generates images from texts, providing a much smoother transition. Check the video and try it by yourself!
Installation
Please see the installation guide.
Getting started
Please see getting started instruction.
Realtime camera demo on GPU
Please see the demo with camera guide.
Continuous txt2img
Please see the demo continuous txt2img.
LICENSE
StreamV2V is licensed under a UT Austin Research LICENSE.
Acknowledgements
Our StreamV2V is highly dependended on the open-source community. Our code is copied and adapted from < StreamDiffusion with LCM-LORA. Besides the base SD 1.5 model, we also use a variaty of LORAs from CIVITAI.
Citing StreamV2V :pray:
If you use StreamV2V in your research or wish to refer to the baseline results published in the paper, please use the following BibTeX entry.
@article{liang2024looking,
title={Looking Backward: Streaming Video-to-Video Translation with Feature Banks},
author={Liang, Feng and Kodaira, Akio and Xu, Chenfeng and Tomizuka, Masayoshi and Keutzer, Kurt and Marculescu, Diana},
journal={arXiv preprint arXiv:2405.15757},
year={2024}
}
@article{kodaira2023streamdiffusion,
title={StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation},
author={Kodaira, Akio and Xu, Chenfeng and Hazama, Toshiki and Yoshimoto, Takanori and Ohno, Kohei and Mitsuhori, Shogo and Sugano, Soichi and Cho, Hanying and Liu, Zhijian and Keutzer, Kurt},
journal={arXiv preprint arXiv:2312.12491},
year={2023}
}