Home

Awesome

OV_SD_CPP

The pure C++ text-to-image pipeline, driven by the OpenVINO native API for Stable Diffusion v1.5 with LMS Discrete Scheduler, supports both static and dynamic model inference. It includes advanced features like Lora integration with safetensors and OpenVINO extension for tokenizer. This demo has been tested on the Windows platform.

Step 1: Prepare environment

C++ Packages:

SD Preparation could be auto implemented with build_dependencies.sh. This script provides 2 ways to install OpenVINO 2023.1.0: conda-forge and Download archives.

cd scripts
chmod +x build_dependencies.sh
./build_dependencies.sh
...
"use conda-forge to install OpenVINO Toolkit 2023.1.0 (C++), or download from archives? (yes/no): "

Notice: Use Intel sample writeOutputBmp function instead of OpenCV for image saving.

Step 2: Prepare SD model and Tokenizer model

SD v1.5 model:

  1. Prepare a conda python env and install dependencies:

    cd scripts
conda create -n SD-CPP python==3.10
pip install -r requirements.txt

  2. Download a huggingface SD v1.5 model like runwayml/stable-diffusion-v1-5, here another model dreamlike-anime-1.0 is used to test for the lora enabling. Ref to the official website for model downloading.

  3. Model conversion from PyTorch model to OpenVINO IR via optimum-intel. Please use the script convert_model.py to convert the model into FP16_static or FP16_dyn, which will be saved into the SD folder.

    cd scripts
python -m convert_model.py -b 1 -t FP16 -sd Path_to_your_SD_model
python -m convert_model.py -b 1 -t FP16 -sd Path_to_your_SD_model -dyn

    Notice: Now the pipeline support batch size = 1 only, ie. static model (1,3,512,512)

Lora enabling with safetensors

Refer this blog for python pipeline, the safetensor model is loaded via src/safetensors.h. The layer name and weight are modified with Eigen Lib and inserted into the SD model with ov::pass::MatcherPass in the file src/lora_cpp.hpp.

SD model dreamlike-anime-1.0 and Lora soulcard are tested in this pipeline. Here, Lora enabling only for FP16.

Download and put safetensors and model IR into the models folder.

Tokenizer model:

3 steps for OpenVINO extension for tokenizer:

  1. The script convert_sd_tokenizer.py in the scripts folder could serialize the tokenizer model IR
  2. Build OV extension: git clone https://github.com/apaniukov/openvino_contrib/ -b tokenizer-fix-decode Refer to PR OpenVINO custom extension ( new feature still in experiments )
  3. Read model with extension in the SD pipeline

Important Notes:

Step 3: Build Pipeline

conda activate SD-CPP
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make

Step 4: Run Pipeline

 ./SD-generate -t <text> -n <negPrompt> -s <seed> --height <output image> --width <output image> -d <debugLogger> -e <useOVExtension> -r <readNPLatent> -m <modelPath> -p <precision> -l <lora.safetensors> -a <alpha> -h <help>

Usage: OV_SD_CPP [OPTION...]

Example:

Positive prompt: cyberpunk cityscape like Tokyo New York with tall buildings at dusk golden hour cinematic lighting

Negative prompt: (empty, here couldn't use OV tokenizer, check the issues for details)

Read the numpy latent instead of C++ std lib for the alignment with Python pipeline

image

image

image

Benchmark:

The performance and image quality of C++ pipeline are aligned with Python

To align the performance with Python SD pipeline, C++ pipeline will print the duration of each model inferencing only

For the diffusion part, the duration is for all the steps of Unet inferencing, which is the bottleneck

For the generation quality, be careful with the negative prompt and random latent generation. C++ random generation with MT19937 results is differ from numpy.random.randn(). Hence, please use -r, --readNPLatent for the alignment with Python(this latent file is for output image 512X512 only)

Program optimization: In addition to inference optimization, now parallel optimization with std::for_each only and add_compile_options(-O3 -march=native -Wall) with CMake

Setup in Windows 10 with VS2019:

  1. Download Anaconda3 and setup Conda env SD-CPP for OpenVINO with conda-forge and use the anaconda prompt terminal for CMake

  2. C++ dependencies:

1. Download from https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip 
2. Unzip to path C:/Eigen3/eigen-3.4.0 
3. Run next step's build.bat will report error: not found Eigen3Config.cmake/eigen3-config.cmake
- Create build folder for Eigen and Open VS in this path C:/Eigen3/eigen-3.4.0/build
- Open VS's developer PS terminal to do "cmake .." and redo the CMake

Ref:not found Eigen3Config.cmake/eigen3-config.cmake

  1. CMake with command lines, create a script build.bat:
rmdir /Q /S build
mkdir build
cd build
cmake -G "Visual Studio 16 2019" -A x64 ^
 -DCMAKE_BUILD_TYPE=Release ^
      ..
cmake --build . --config Release

  1. Put safetensors and model IR into the models folder with the following default path: models\dreamlike-anime-1.0\FP16_static models\soulcard.safetensors

  2. Run with prompt:

cd PROJECT_SOURCE_DIR\build
.\Release\SD-generate.exe -l ''  // without lora
.\Release\SD-generate.exe -l ../models/soulcard.safetensors

Notice: 
* must run command line within path of build, or .exe could not find the models
* .exe is in the Release folder
  1. Debug within Visual Studio(open .sln file in the build folder)

Notice: has issue to build OpenVINO custom extension on Windows platform, so use the default tokenizer.