Awesome
<p align="center"> <img src="./docs/logo.png" height=150> </p>Story-Adapter: A Training-free Iterative Framework for Long Story Visualization
<span> <a href="https://arxiv.org/abs/2410.06244"><img src="https://img.shields.io/badge/arXiv-2410.06244-b31b1b.svg" height=22.5></a> <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" height=22.5></a> <a href="https://jwmao1.github.io/storyadapter/"><img src="https://img.shields.io/badge/project-StoryAdapter-purple.svg" height=22.5></a> </span>Code for the paper Story-Adapter: A Training-free Iterative Framework for Long Story Visualization
Note: This code base is still not complete.
About this repo:
The repository contains the official implementation of "Story-Adapter".
Introduction 🦖
<br> <img src="./docs/teaser-github.jpg" width="800"/>Story visualization, the task of generating coherent images based on a narrative, has seen significant advancements with the emergence of text-to-image models, particularly diffusion models. However, maintaining semantic consistency, generating high-quality fine-grained interactions, and ensuring computational feasibility remain challenging, especially in long story visualization (i.e., up to 100 frames). In this work, we propose a training-free and computationally efficient framework, termed Story-Adapter, to enhance the generative capability of long stories. Specifically, we propose an iterative paradigm to refine each generated image, leveraging both the text prompt and all generated images from the previous iteration. Central to our framework is a training-free global reference cross-attention module, which aggregates all generated images from the previous iteration to preserve semantic consistency across the entire story, while minimizing computational costs with global embeddings. This iterative process progressively optimizes image generation by repeatedly incorporating text constraints, resulting in more precise and fine-grained interactions. Extensive experiments validate the superiority of Story-Adapter in improving both semantic consistency and generative capability for fine-grained interactions, particularly in long story scenarios.
News 🚀
- 2024.10.10: Paper is released on ArXiv.
- 2024.10.04: Code released.
Framework 🤖
<br> <img src="./docs/framework.jpg" width="1080"/>Story-Adapter framework. Illustration of the proposed iterative paradigm, which consists of initialization, iterations in Story-Adapter, and implementation of Global Reference Cross-Attention (GRCA). Story-Adapter first visualizes each image only based on the text prompt of the story and uses all results as reference images for the future round. In the iterative paradigm, Story-Adapter inserts GRCA into SD. For the ith iteration of each image visualization, GRCA will aggregate the information flow of all reference images during the denoising process through cross-attention. All results from this iteration will be used as a reference image to guide the dynamic update of the story visualization in the next iteration.
Quick Start 🔧
Installation
The project is built with Python 3.10.14, PyTorch 2.2.2. CUDA 12.1, cuDNN 8.9.02 For installing, follow these instructions:
# git clone this repository
git clone https://github.com/jwmao1/story-adapter.git
cd story-adapter
# create new anaconda env
conda create -n StoryAdapter python=3.10
conda activate StoryAdapter
# install packages
pip install -r requirements.txt
Download the checkpoint
- downloading RealVisXL_V4.0 put it into "./RealVisXL_V4.0"
- downloading clip_image_encoder put it into "./IP-Adapter/sdxl_models/image_encoder"
- downloading ip-adapter_sdxl put it into "./IP-Adapter/sdxl_models/ip-adapter_sdxl.bin"
Running Demo
python run.py --base_model_path your_path/RealVisXL_V4.0 --image_encoder_path your_path/IP-Adapter/sdxl_models/image_encoder --ip_ckpt your_path//IP-Adapter/sdxl_models/ip-adapter_sdxl.bin
Customized Running
python run.py --base_model_path your_path/RealVisXL_V4.0 --image_encoder_path your_path/IP-Adapter/sdxl_models/image_encoder --ip_ckpt your_path//IP-Adapter/sdxl_models/ip-adapter_sdxl.bin
--story [your story]
Performance 🎨
Regular-length Story Visualization
- downloading the StorySalon test set."
| GIF1 | GIF2 | GIF3 |
|---|---|---|
| <img src="./docs/our_005169.gif" alt="GIF 1" width="224"/> | <img src="./docs/our_007016.gif" alt="GIF 2" width="224"/> | <img src="./docs/our_007137.gif" alt="GIF 3" width="224"/> |
| GIF4 | GIF5 | GIF6 |
|---|---|---|
| <img src="./docs/our_013804.gif" alt="GIF 4" width="224"/> | <img src="./docs/our_015770.gif" alt="GIF 5" width="224"/> | <img src="./docs/our_000026.gif" alt="GIF 6" width="224"/> |
| GIF7 | GIF8 | GIF9 |
|---|---|---|
| <img src="./docs/our_012060.gif" alt="GIF 7" width="224"/> | <img src="./docs/our_008614.gif" alt="GIF 8" width="224"/> | <img src="./docs/our_008710.gif" alt="GIF 9" width="224"/> |
Long Story Visualization
<br> <img src="./docs/comic1.png" width="1080"/> <br> <br> <img src="./docs/comic7.png" width="1080"/> <br> <br> <img src="./docs/comic3.png" width="1080"/>Acknowledgement 🍻
Deeply appreciate these wonderful open source projects: stablediffusion, clip, ip-adapter, storygen, storydiffusion, timm.
Citation 🔖
If you find this repository useful, please consider giving a star ⭐ and citation 🙈:
@misc{mao2024story_adapter,
title={{Story-Adapter: A Training-free Iterative Framework for Long Story Visualization}},
author={Mao, Jiawei and Huang, Xiaoke and Xie, Yunfei and Chang, Yuanqi and Hui, Mude and Xu, Bingjie and Zhou, Yuyin},
journal={arXiv},
volume={abs/2410.06244},
year={2024},
}