Home

Awesome

<h1 align="center"> SignGen: End-to-End Sign Language Video Generation with Latent Diffusion</h1>

Method

<img src="pic/framework_10.png" alt="Editor" width="900">

Experiment Results

RWTH-2014

case1case2case3
case4case4case4
case4case4case4
case4case4case4

RWTH-2014T

case1case2case3
case4case4case4
case4case4case4

AUTSL

case4

case4

Running by Yourself

1. Installation

create a conda environment.

conda create -n  xxx  python==3.8.5

Then you can create the same environment as ours with the following command:

 pip install -r requirements.txt # install all requirements

2. Download model weights

For LPIPS

The code will do it for you!

Code will download Alex and move it into: models/weights/v0.1/alex.pth

For FVD

The code will do it for you!

Code will download i3D model pretrained on Kinetics-400 Use models/fvd/convert_tf_pretrained.py to make i3d_pretrained_400.pt

3. Datasets

You can download these datasets such as RWTH-2014,RWTH-2014T and AUTSL.

How the data was processed:

  1. Download AUTSL dataset to /path/to/AUTSL:\
  2. Convert 128x128 images to HDF5 format:
    `python datasets/sign_language_convert.py --sl_dir 'datasets/videos' --split 'train' --out_dir 'datasets/signLanguages/train' --image_size 128 --force_h5 False

Training -The code is coming soon.