Awesome

MSDTGP (IEEE TGRS 2022)

📖Paper | 🖼️PDF | 🎁Dataset

PyTorch codes for "Satellite Video Super-resolution via Multi-Scale Deformable Convolution Alignment and Temporal Grouping Projection", IEEE Transactions on Geoscience and Remote Sensing (TGRS), 2022.

Authors: Yi Xiao, Xin Su, Qiangqiang Yuan*, Denghong Liu, Huanfeng Shen, and Liangpei Zhang<br> Wuhan University

Abstract

As a new earth observation tool, satellite video has been widely used in remote-sensing field for dynamic analysis. Video super-resolution (VSR) technique has thus attracted increasing attention due to its improvement to spatial resolution of satellite video. However, the difficulty of remote-sensing image alignment and the low efficiency of spatial–temporal information fusion make poor generalization of the conventional VSR methods applied to satellite videos. In this article, a novel fusion strategy of temporal grouping projection and an accurate alignment module are proposed for satellite VSR. First, we propose a deformable convolution alignment module with a multiscale residual block to alleviate the alignment difficulties caused by scarce motion and various scales of moving objects in remote-sensing images. Second, a temporal grouping projection fusion strategy is proposed, which can reduce the complexity of projection and make the spatial features of reference frames play a continuous guiding role in spatial–temporal information fusion. Finally, a temporal attention module is designed to adaptively learn the different contributions of temporal information extracted from each group. Extensive experiments on Jilin-1 satellite video demonstrate that our method is superior to current state-of-the-art VSR methods.

Network

🧩Install

git clone https://github.com/XY-boy/MSDTGP.git

Environment

• CUDA 10.0
• pytorch 1.x
• build DCNv2

Dataset Preparation

Please download our dataset in

• Baidu Netdisk Jilin-189 Code:31ct
• Zenodo: <a href="https://doi.org/10.5281/zenodo.6969604"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.6969604.svg" alt="DOI"></a>

You can also train your dataset following the directory sturture below!

Data directory structure

trainset--
 | train--
  | LR4x---
   | 000.png
   | ···.png
   | 099.png
  | GT---
  | Bicubic4x---

testset--
 | eval--
  | LR4x---
   | 000.png
   | ···.png
   | 099.png
  | GT---
  | Bicubic4x---

Training

python main.py

Test

python eval.py

Quantitative results

Qualitative results

More details can be found in our paper!

Contact

If you have any questions or suggestions, feel free to contact me. 😊
Email: xiao_yi@whu.edu.cn; xy574475@gmail.com

Citation

If you find our work helpful in your research, please consider citing it. Thank you! 😊😊

@ARTICLE{xiao2022msdtgp,  
author={Xiao, Yi and Su, Xin and Yuan, Qiangqiang and Liu, Denghong and Shen, Huanfeng and Zhang, Liangpei},  
journal={IEEE Transactions on Geoscience and Remote Sensing},  
title={Satellite Video Super-Resolution via Multiscale Deformable Convolution Alignment and Temporal Grouping Projection},   
year={2022},  
volume={60},  
number={},  
pages={1-19},  
doi={10.1109/TGRS.2021.3107352}
}

Acknowledgement

Our work is built upon RBPN and TDAN.
Thanks to the author for the source code !