Awesome

Video Restoration Framework and its Meta-adaptations to Data-poor Conditions (ECCV2022)

Prashant W Patil, Sunil Gupta, Santu Rana, and Svetha Venkatesh

Abstract: Restoration of weather degraded videos is a challenging problem due to diverse weather conditions e.g., rain, haze, snow, etc. Existing works handle video restoration for each weather using a different custom-designed architecture. This approach has many limitations. First, a custom-designed architecture for each weather condition requires domain-specific knowledge. Second, disparate network architectures across weather conditions prevent easy knowledge transfer to novel weather conditions where we do not have a lot of data to train a model from scratch. For example, while there is a lot of common knowledge to exploit between the models of different weather conditions at day or night time, it is difficult to do such adaptation. To this end, we propose a generic architecture that is effective for any weather condition due to the ability to extract robust feature maps without any domain-specific knowledge. This is achieved by novel components: spatio-temporal feature modulation, multi-level feature aggregation, and recurrent guidance decoder. Next, we propose a meta-learning based adaptation of our deep architecture to the restoration of videos in data-poor conditions (night-time videos). We show comprehensive results on video de-hazing and de-raining datasets in addition to the meta-learning based adaptation results on night-time video restoration tasks. Our results clearly outperform the state-of-theart weather degraded video restoration methods.

Network Architecture

Requirements:

Python >= 3.5

Tensorflow == 2.0

Numpy

PIL

Testing Videos:

Keep Testing Videos Frames in "videos/{dataset}" folder.

Checkpoints:

The checkpoints are provided for:
1. Scratch trained checkpoints for REVIDE and RainSynAll100 datasets.
2. Meta adapted checkpoints for "night time" Haze, Rain, and Rain+veling dataset.
3. Keep the checkpoints in "./checkpoints/dataset/"

Download the checkpoint:

Testing Procedure:

1. select options  --dataset, --test_dir, --checkpoint_path in "options.py"  
2. Run "testing.py"
3. Results will be saved in --output_path

Database:

Synthetically Generated Night-time Weather Degraded Database is available at:

## Citation
If our method is useful for your research, please consider citing:
    
@inproceedings{patil2022video,
  title={Video Restoration Framework and Its Meta-adaptations to Data-Poor Conditions},
  author={Patil, Prashant W and Gupta, Sunil and Rana, Santu and Venkatesh, Svetha},
  booktitle={European Conference on Computer Vision},
  pages={143--160},
  year={2022},
  organization={Springer}
}


## Contact
Please contact prashant.patil@deakin.edu.au, if you are facing any issue.