Awesome

STFAN

Code repo for the paper "Spatio-Temporal Filter Adaptive Network for Video Deblurring". [Paper] [Project Page]

Filter Adaptive Convolutional (FAC) Layer

The proposed filter adaptive convolutional (FAC) layer applies generated pixel variant filters to the features, In theory, the element-wise adaptive filters is five-dimensional $h{\times}w{\times}c{\times}k{\times}k$ . In practice, the dimension of the generated filter $\mathcal{F}$ is $h{\times}w{\times}ck^2$ and we reshape it into the five-dimensional filter. For each position $(x,y,c_i)$ of input feature $Q\in\mathds{R}^{h{\times}w{\times}c}$ , a specific local filter $\mathds{F}_{x,y,c_i}\in\mathcal{R}^{k{\times}k}$ (reshape from $1{\times}1{\times}k^2$ ) is applied to the region centered around $Q_{x,y,c_i}$ .

<img width=40% src="https://user-images.githubusercontent.com/14334509/68988034-36f90100-086c-11ea-9e57-93aca737d6d3.jpg"> ➤ The forward pass of the proposed Filter Adaptive Convolutional (FAC) Layer is as follows: <img width=70% src="https://user-images.githubusercontent.com/14334509/68988093-1aa99400-086d-11ea-9a92-8cef3026b11a.png"> ➤ The backward pass can be presented as: <img width=65% src="https://user-images.githubusercontent.com/14334509/68988094-2006de80-086d-11ea-9be0-d9758be6380e.png"> <img width=50% src="https://user-images.githubusercontent.com/14334509/68988097-22693880-086d-11ea-983e-c561b7421d26.png">

Illustration of Alignment and Deblurring Processes by FAC layer

The frame alignment and deblurring are both spatially variant tasks. Using the proposed FAC layer, we consider these two processes as two filter adaptive convolution in feature domain. The convolution operation can transform the pixels of features, which can be used for frames alignment (a) and deblurring (b) using estimated corresponding filters.

Pretrained Models

You could download the pretrained model (21.5MB) of STFAN from here.

(Note that the model does not need to unzip, just load it directly.)

Prerequisites

Linux (tested on Ubuntu 14.04/16.04)
CUDA 8.0/9.0/10.0
gcc 4.9+
Python 2.7+
PyTorch 1.0+
easydict
tensorboardX Note that if your CUDA is 10.2+, you need to modify KernelConv2D_cuda.cpp:

1. Uncomment: #include <c10/cuda/CUDAStream.h>
2. Modify: at::cuda::getCurrentCUDAStream() -> c10::cuda::getCurrentCUDAStream()

Installation

pip install -r requirements.txt
bash install.sh

Get Started

Download DeepVideoDeblurring dataset from this link, and modify the data index tree as following:

├── [DATASET_ROOT]
│   ├── train
│   │   ├── input
│   │   ├── GT
│   ├── test
│   │   ├── input
│   │   ├── GT

Use the following command to train the neural network:

python runner.py 
        --phase 'train'\
        --data [dataset path]\
        --out [output path]

Use the following command to test the neural network:

python runner.py \
        --phase 'test'\
        --weights './ckpt/best-ckpt.pth.tar'\
        --data [dataset path]\
        --out [output path]

Use the following command to resume training the neural network:

python runner.py 
        --phase 'resume'\
        --weights './ckpt/best-ckpt.pth.tar'\
        --data [dataset path]\
        --out [output path]

You can also use the following simple command, with changing the settings in config.py:

python runner.py

Results on the testing dataset and real blurry videos

Some results are shown in [Project Page].

Citation

If you find STFANet, or FAC layer useful in your research, please consider citing:

@inproceedings{zhou2019stfan,
  title={Spatio-Temporal Filter Adaptive Network for Video Deblurring},
  author={Zhou, Shangchen and Zhang, Jiawei and Pan, Jinshan and Xie, Haozhe and  Zuo, Wangmeng and Ren, Jimmy},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  year={2019}
}

Contact

We are glad to hear if you have any suggestions and questions.

Please send email to shangchenzhou@gmail.com

License

This project is open sourced under MIT license.