Awesome

MMSR

PyTorch implementation of "Learning Mutual Modulation for Self-Supervised Cross-Modal Super-Resolution"
[ECCV paper, supp] [arXiv]

Updates

[2022/07/04] Paper was accepted by ECCV2022. We thank anonymous reviewers from CVPR2022 and ECCV2022 for their suggestions to our paper. See you in Tel-Aviv!
[2022/07/19] Paper is available on arXiv.
[2022/07/20] Code was uploaded. Our code was built on the repository of P2P (ICCV2019), we thank the authors for their efforts.

Abstract

Self-supervised cross-modal super-resolution (SR) can overcome the difficulty of acquiring paired training data, but is challenging because only low-resolution (LR) source and high-resolution (HR) guide images from different modalities are available.
Existing methods utilize pseudo or weak supervision in LR space and thus deliver results that are blurry or not faithful to the source modality.

To address this issue, we present a mutual modulation SR (MMSR) model (Fig. 2), which tackles the task by a mutual modulation strategy, including a source-to-guide modulation (Fig. 3) and a guide-to-source modulation (Fig. 4).
In these modulations, we develop cross-domain adaptive filters to fully exploit cross-modal spatial dependency and help induce the source to emulate the resolution of the guide and induce the guide to mimic the modality characteristics of the source (Fig. 9).
Moreover, we adopt a cycle consistency constraint to train MMSR in a fully self-supervised manner. Experiments on various tasks demonstrate the state-of-the-art performance of our MMSR (Figs. 1, IV, 11, 12).

Quick Start

Please check inference.ipynb.
For any question or discussion, reach Xiaoyu Dong at dong@ms.k.u-tokyo.ac.jp.

Citation

@InProceedings{Dong2022MMSR,
  author    = {Dong, Xiaoyu and Yokoya, Naoto and Wang, Longguang and Uezato, Tatsumi},
  title     = {Learning Mutual Modulation for Self-Supervised Cross-Modal Super-Resolution},
  booktitle = {ECCV},
  year      = {2022}
}