Awesome

Deep Video Deblurring: The Devil is in the Details

This PyTorch implementation accompanies the paper:

Jochen Gast and Stefan Roth, Deep Video Deblurring: The Devil is in the Details,<br> ICCV Workshop on Learning for Computational Imaging (ICCVW), Seoul, Korea, November, 2019

Preprint: https://arxiv.org/abs/1909.12196 <a href="https://www.visinf.tu-darmstadt.de"> <img align="right" src="https://github.com/ezjong/deepvisinf/blob/deblur-devil-release/resources/vi-logo-small.jpg" alt="DeepVisinf Logo" height="55"/> </a> <br> Contact: Jochen Gast (jochen.gast@visinf.tu-darmstadt.de) <p align="right"> <a href="https://www.visinf.tu-darmstadt.de">Visual Inference</a> </p>

• Components for Deep Video Deblurring
• Datasets
• Installation, External Packages & Telegram Bot
• Deep Visual Inference Framework
  • Adding Classes
  • Everything is a Dictionary
  • Magic Keys
  • Data Loader Collation
  • Logging Facilities
  • Custom Tensorboard Calls
  • More Settings
• License
• Citation

Components for Deep Video Deblurring

Ignoring the framework code, the components for Deep Video Deblurring are contained in the modules

Datasets

The mirrors of the GoPro datasets can be found here:

Installation, External Packages & Telegram Bot

This code has been tested with Python 3.6, PyTorch 1.2, Cuda 10.0 but is likely to run with newer versions of PyTorch and Cuda. For installing PyTorch, follow the steps here.

Additional dependencies can be installed by running the script

pip install -r requirements.txt

The code for the PWC network relies on a correlation layer implementation provided by Clément Pinard.<br> To install the correlation layer, run

cd external_packages && . install_spatial_correlation_sampler.sh

The framework supports status updates sent via Telegram, i.e. using

logging.telegram(msg, *args)

Per default, any finished epoch will send an update with the current losses. Note that the dependencies for the Telegram bot, i.e. filetype jsonpickle telegrambotapiwrapper, are included in requirements.txt, but they are not essential to the framework.

In order to use the Telegram bot, you need to copy dot_machines.json to ~/.machines.json and fill in the required chat_id and tokens. Here you can find out how to obtain these credentials.

Scripts and Models

The scripts make use of the environment variables DATASETS_HOME pointing to the root folder of the respective dataset, and EXPERIMENTS_HOME to create output artifacts.

To train the models you need the prewarping networks. Place them in $EXPERIMENTS_HOME/prewarping_networks.

More models will follow soon!

Deep Visual Inference Framework

Adding Classes

For adding new classes/functions following packages are relevant:

Everything is a Dictionary

Most things are automated for convenience. To accomodate this automation, there are three essential Python dictionaries being passed around which hold any artifact produced along the way

Keep this in mind, when adding new functionality.

Magic Keys

Again, the framework passes around dictionaries between augmentations, datasets, losses, models, and visualizers. You may want a subset of these tensors to be transfered to Cuda along the way. The are two magic prefixes for dictionary keys to assure that, i.e. input* and target*. Any input or target tensor that should be transfered to Cuda along the way, has to start with one of these prefixes.

E.g. a dataset may return a dictionary with the keys input1 and target1 (both cpu tensors); any augmentation, loss, or model will fetch these as gpu tensors.

Data Loader Collation

This is in particular interesting for the GoPro datasets. The framework allows dataloaders to return multiple -- typically photometric -- augmentations per single example, in order to increase throughput when loading large images. In such a case a tensor returned from a single dataset example should have three dimensions. The pipeline will stack the first dimension into a batch.

Let's say, a dataset returns four examples per load, i.e. input1 and target1 tensors have the dimension 4xHxW. For, e.g. batch_size=8, the resulting tensors (received by the augmentation, loss, model, and visualizers) will be 32xHxW.

Logging Facilities

There are some automated logging facilities which log any output into the given save directory.

• A args.txt and args.json file remembering the passed arguments.
• A logbook.txt gets created remembering all calls to logging.info(msg, *args).
• CSV/MAT file generation of any loss (of any epoch) found in the loss_dict.
• For every run, the complete source code gets zipped into the log directory.
• There is always a tensorboard created at the $SAVE/tb directory.

Custom Tensorboard Calls

When calling tensorboard summaries in your models for debugging, I suggest to replace any call to torch.utils.tensorboard.SummaryWriter.add_* by utils.summary.*

For instance, instead of

from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter(..)
writer.add_scalar(tensor1)
writer.add_images(tensor2)

use

from utils import summary
summary.scalar(tensor1)
summary.images(tensor2)

Advantages:

• This will automatically use the created tensorboard in the save directory.
• The current epoch gets automatically set by the framework.
• Otherwise, they are identical.

More Settings

A limited amount of customization is available in constants.py where you can set

• Logging indents, time zones, and colors
• Filename defaults
• Flush intervals
• etc

License

This code is licensed under the Apache License, Version 2.0, as found in the LICENSE file.

Citation

# brief
@inproceedings{Gast:2019:DVD,
    Author = {Jochen Gast and Stefan Roth},
    Booktitle = {ICCV Workshops},
    Title = {Deep Video Deblurring: {T}he Devil is in the Details},
    Year = {2019}
}

# detailed
@inproceedings{Gast:2019:DVD,
    Address = {Seoul, Korea},
    Author = {Jochen Gast and Stefan Roth},
    Booktitle = {ICCV Workshop on Learning for Computational Imaging (ICCVW)},
    Month = nov,
    Title = {Deep Video Deblurring: {T}he Devil is in the Details},
    Year = {2019}
}