Awesome
LDMVFI: Video Frame Interpolation with Latent Diffusion Models
Duolikun Danier, Fan Zhang, David Bull
Overview
We observe that most existing learning-based VFI models are trained to minimise the L1/L2/VGG loss between their outputs and the ground-truth frames. However, it was shown in previous works that these metrics do not correlate well with the perceptual quality of VFI. On the other hand, generative models, especially diffusion models, are showing remarkable results in generating visual content with high perceptual quality. In this work, we leverage the high-fidelity image/video generation capabilities of latent diffusion models to perform generative VFI.
<p align="center"> <img src="https://danier97.github.io/LDMVFI/overall.svg" alt="Paper" width="60%"> </p>Dependencies and Installation
See environment.yaml for requirements on packages. Simple installation:
conda env create -f environment.yaml
Pre-trained Model
The pre-trained model can be downloaded from here, and its corresponding config file is this yaml.
Preparing datasets
Training sets:
[Vimeo-90K] | [BVI-DVC quintuplets]
Test sets:
[Middlebury] | [UCF101] | [DAVIS] | [SNU-FILM]
To make use of the evaluate.py and the files in ldm/data/, the dataset folder names should be lower-case and structured as follows.
└──── <data directory>/
├──── middlebury_others/
| ├──── input/
| | ├──── Beanbags/
| | ├──── ...
| | └──── Walking/
| └──── gt/
| ├──── Beanbags/
| ├──── ...
| └──── Walking/
├──── ucf101/
| ├──── 0/
| ├──── ...
| └──── 99/
├──── davis90/
| ├──── bear/
| ├──── ...
| └──── walking/
├──── snufilm/
| ├──── test-easy.txt
| ├──── ...
| └──── data/SNU-FILM/test/...
├──── bvidvc/quintuplets
| ├──── 00000/
| ├──── ...
| └──── 17599/
└──── vimeo_septuplet/
├──── sequences/
├──── sep_testlist.txt
└──── sep_trainlist.txt
Evaluation
To evaluate LDMVFI (with DDIM sampler), for example, on the Middlebury dataset, using PSNR/SSIM/LPIPS, run the following command.
python evaluate.py \
--config configs/ldm/ldmvfi-vqflow-f32-c256-concat_max.yaml \
--ckpt <path/to/ldmvfi-vqflow-f32-c256-concat_max.ckpt> \
--dataset Middlebury_others \
--metrics PSNR SSIM LPIPS \
--data_dir <path/to/data/dir> \
--out_dir eval_results/ldmvfi-vqflow-f32-c256-concat_max/ \
--use_ddim
This will create the directory eval_results/ldmvfi-vqflow-f32-c256-concat_max/Middlebury_others/
, and store the interpolated frames, as well as a results.txt
file in that directory. For other test sets, replace Middlebury_other
with the corresponding class names defined in ldm/data/testsets.py (e.g. Ucf101_triplet
).
To evaluate the model on perceptual video metric FloLPIPS, first evaluate the image metrics using the code above (so that the interpolated frames are saved in eval_results/ldmvfi-vqflow-f32-c256-concat_max
), then run the following code.
python evaluate_vqm.py \
--exp ldmvfi-vqflow-f32-c256-concat_max \
--dataset Middlebury_others \
--metrics FloLPIPS \
--data_dir <path/to/data/dir> \
--out_dir eval_results/ldmvfi-vqflow-f32-c256-concat_max/ \
This will read the interpolated frames previously stored in eval_results/ldmvfi-vqflow-f32-c256-concat_max/Middlebury_others/
then output the evaluation results to results_vqm.txt
in the same folder.
To interpolate a video (in .yuv format), use the following code.
python interpolate_yuv.py \
--net LDMVFI \
--config configs/ldm/ldmvfi-vqflow-f32-c256-concat_max.yaml \
--ckpt <path/to/ldmvfi-vqflow-f32-c256-concat_max.ckpt> \
--input_yuv <path/to/input/yuv> \
--size <spatial res of video, e.g. 1920x1080> \
--out_fps <output fps, should be 2 x original fps> \
--out_dir <desired/output/dir> \
--use_ddim
Training
LDMVFI is trained in two stages, where the VQ-FIGAN and the denoising U-Net are trained separately.
VQ-FIGAN
python main.py --base configs/autoencoder/vqflow-f32.yaml -t --gpus 0,
Denoising U-Net
python main.py --base configs/ldm/ldmvfi-vqflow-f32-c256-concat_max.yaml -t --gpus 0,
These will create a logs/
folder within which the corresonding directories are created for each experiment. The log files from training include checkpoints, images and tensorboard loggings.
To resume from a checkpoint file, simply use the --resume
argument in main.py to specify the checkpoint.
Citation
@article{danier2023ldmvfi,
title={LDMVFI: Video Frame Interpolation with Latent Diffusion Models},
author={Danier, Duolikun and Zhang, Fan and Bull, David},
journal={arXiv preprint arXiv:2303.09508},
year={2023}
}
Acknowledgement
Our code is adapted from the original latent-diffusion repository. We thank the authors for sharing their code.