Awesome

Stable Diffusion in TensorFlow / Keras

A Keras / Tensorflow implementation of Stable Diffusion.

The weights were ported from the original implementation.

Colab Notebooks

The easiest way to try it out is to use one of the Colab notebooks:

GPU Colab
GPU Colab Img2Img
GPU Colab Inpainting
GPU Colab - Tile / Texture generation
GPU Colab - Loading Pytorch ckpt Weights
GPU Colab + Mixed Precision
- ~10s generation time per image (512x512) on default Colab GPU without drop in quality (source)
TPU Colab.
- Slower than GPU for single-image generation, faster for large batch of 8+ images (source).
GPU Colab with Gradio
GPU Colab - Video Generation

Installation

Install as a python package

Install using pip with the git repo:

pip install git+https://github.com/divamgupta/stable-diffusion-tensorflow

Installing using the repo

Download the repo, either by downloading the zip file or by cloning the repo with git:

git clone git@github.com:divamgupta/stable-diffusion-tensorflow.git

Using pip without a virtual environment

Install dependencies using the requirements.txt file or the requirements_m1.txt file,:

pip install -r requirements.txt

Using a virtual environment with virtualenv

Create your virtual environment for python3:
```
python3 -m venv venv
```
Activate your virtualenv:
```
source venv/bin/activate
```
Install dependencies using the requirements.txt file or the requirements_m1.txt file,:
```
pip install -r requirements.txt
```

Usage

Using the Python interface

If you installed the package, you can use it as follows:

from stable_diffusion_tf.stable_diffusion import StableDiffusion
from PIL import Image

generator = StableDiffusion(
    img_height=512,
    img_width=512,
    jit_compile=False,
)
img = generator.generate(
    "An astronaut riding a horse",
    num_steps=50,
    unconditional_guidance_scale=7.5,
    temperature=1,
    batch_size=1,
)

# for image to image :
img = generator.generate(
    "A Halloween bedroom",
    num_steps=50,
    unconditional_guidance_scale=7.5,
    temperature=1,
    batch_size=1,
    input_image="/path/to/img.png"
)


Image.fromarray(img[0]).save("output.png")

Using `text2image.py` from the git repo

Assuming you have installed the required packages, you can generate images from a text prompt using:

python text2image.py --prompt="An astronaut riding a horse"

The generated image will be named output.png on the root of the repo. If you want to use a different name, use the --output flag.

python text2image.py --prompt="An astronaut riding a horse" --output="my_image.png"

Check out the text2image.py file for more options, including image size, number of steps, etc.

Using `img2img.py` from the git repo

Assuming you have installed the required packages, you can modify images from a text prompt using:

python img2img.py --prompt="a high quality sketch of people standing with sun and grass , watercolor , pencil color" --input="img.jpeg"

The generated image will be named img2img-out.jpeg by default on the root of the repo. If you want to use a different name, use the --output flag.

Check out the img2img.py file for more options, including the number of steps.

Example outputs

The following outputs have been generated using this implementation:

A epic and beautiful rococo werewolf drinking coffee, in a burning coffee shop. ultra-detailed. anime, pixiv, uhd 8k cryengine, octane render

Spider-Gwen Gwen-Stacy Skyscraper Pink White Pink-White Spiderman Photo-realistic 4K

A vision of paradise, Unreal Engine

Inpainting

Image2Image

a high quality sketch of people standing with sun and grass , watercolor , pencil color <img width="884" alt="Screen Shot 2022-10-09 at 9 34 30 AM" src="https://user-images.githubusercontent.com/1890549/194768637-f586772d-aef5-4d64-8dd5-f7f4962924e1.png">

Keras Stable Diffusion Video Generation

A beautiful street view of prague, artstation concept art, extremely detailed oil painting, vivid colors

https://user-images.githubusercontent.com/63783894/201447745-6a3a96f4-f065-4e54-be5d-01941475a31c.mp4