Awesome

基于非同源数据的SAR图像生成方法研究（Research on SAR image generation method based on non-homologous data）

一、Introduction

合成孔径雷达作为遥感领域的研究重点之一，不仅在民用方面应用广泛，在军事应用领域上也发挥着越来越大的作用。由于合成孔径雷达具有全天侯、全天时、多视角、多分辨率数据获取的能力，对合成孔径雷达的研究已成为图像数据获取、处理研究发展的“热点”问题。现有方法解决SAR图像处理问题时，常借助辅助数据或辅助信息对原有的数据集进行数据扩充或特征增强。生成对抗网络（GAN）是一种灵活高效的生成模型，可以学习真实数据的分布。循环生成对抗网络（CycleGAN）是GAN的一种模型，它可以学习将图像从源域X转向目标域Y，并且不需要成对的图片作为训练数据，虽然该模型生成的图像质量不如基于条件生成对抗网络的有监督学习模型（Pix2pixGAN），但其应用场景更加灵活。本研究方法在对光学图像和SAR图像特征的异同进行比较并且对不同的生成对抗网络模型进行比较后，采用光学图像作为他源信息辅助，利用非同源的图像转换技术，采用循环生成对抗网络模型，将光学目标图像转换为SAR目标图像，并且对生成图像进行评估量化方案的设计，使最终生成数据的识别性能能在SAR标准数据的基础上有一定程度的提高。

Synthetic aperture radar, as one of the research focuses in the field of remote sensing, is not only widely used in civilian applications, but also plays an increasingly important role in military applications. Due to the ability of synthetic aperture radar to acquire all-weather, all-day, multi-view and multi-resolution data, the research on synthetic aperture radar has become a "hot" issue in the development of image data acquisition and processing. When the existing methods solve the problem of SAR image processing, they often use auxiliary data or auxiliary information to perform data expansion or feature enhancement on the original data set. Generative Adversarial Networks (GANs) are flexible and efficient generative models that can learn the distribution of real data. The Cyclic Generative Adversarial Network (CycleGAN) is a model of GAN that can learn to transfer images from the source domain X to the target domain Y, and does not require paired images as training data, although the image quality generated by this model is not as good as conditional generation. A supervised learning model for adversarial networks (Pix2pixGAN), but its application scenarios are more flexible. In this research method, after comparing the similarities and differences of optical image and SAR image features and comparing different generative adversarial network models, the optical image is used as the aid of other source information, the non-homologous image conversion technology is used, and the recurrent generative adversarial network is used. Model, convert the optical target image into SAR target image, and design the quantification scheme for the evaluation of the generated image, so that the recognition performance of the final generated data can be improved to a certain extent based on the SAR standard data.

二、Experiment procedure

搭建环境

1、安装Anaconda

2、下载CycleGAN-pytorch源码，下载地址：CycleGAN

3、运行如下代码创建虚拟环境

cd pytorch-CycleGAN-and-pix2pix
conda env create -f environment.yml

4、关于visdom无法成功安装的问题在虚拟环境下运行代码

pip install visdom
python -m visdom.server

首次安装时间过长导致安装失败的解决办法：

①进入Anaconda的目录：D:\anaconda3\Lib\site-packages\visdom

②替换其中的static文件夹

③进入python文件server，将第1917行download_scripts()注释掉

visdom安装失败截图： <img src=./images/visdom-fail.png>

CycleGAN部分

数据集

1、数据集结构

datasets---opt2sar---trainA
                  ---trainB
                  ---testA
                  ---testB

2、数据集准备光学数据集：LEVIR SAR数据集：SAR-ship-detection

也可以使用其他数据集光学数据集汇总：optical SAR数据集汇总：SAR

使用CycleGAN进行训练和测试

运行如下命令进行训练： cd C:\Users\amin\Desktop\pytorch-CycleGAN-and-pix2pix-master conda activate pytorch-CycleGAN-and-pix2pix python -m visdom.server python train.py --dataroot ./datasets/opt2sar --name opt2sar_cyclegan --model cycle_gan

中断训练后继续训练： python train.py --dataroot ./datasets/opt2sar --name opt2sar_cyclegan --model cycle_gan --continue_train --epoch_count 4

测试： python test.py --dataroot ./datasets/opt2sar --name opt2sar_cyclegan --model cycle_gan

关闭环境： conda deactivate

一些建议：tips

训练过程截图： <img src=./images/cyclegan-train-success.png>

参数的修改

example： 1、修改lambdaA、lambdaB 进入models/cycle_gan_model.py找到如下代码进行修改：

parser.set_defaults(no_dropout=True)  # default CycleGAN did not use dropout
        if is_train:
            parser.add_argument('--lambda_A', type=float, default=10.0, help='weight for cycle loss (A -> B -> A)')
            parser.add_argument('--lambda_B', type=float, default=10.0, help='weight for cycle loss (B -> A -> B)')
            parser.add_argument('--lambda_identity', type=float, default=0.5, help='use identity mapping. Setting lambda_identity other than 0 has an effect of scaling the weight of the identity mapping loss. For example, if the weight of the identity loss should be 10 times smaller than the weight of the reconstruction loss, please set lambda_identity = 0.1')

        return parser

2、修改epoch、learning rate等进入options/train_options.py,找到如下代码进行修改：

def initialize(self, parser):
        parser = BaseOptions.initialize(self, parser)
        # visdom and HTML visualization parameters
        parser.add_argument('--display_freq', type=int, default=400, help='frequency of showing training results on screen')
        parser.add_argument('--display_ncols', type=int, default=4, help='if positive, display all images in a single visdom web panel with certain number of images per row.')
        parser.add_argument('--display_id', type=int, default=1, help='window id of the web display')
        parser.add_argument('--display_server', type=str, default="http://localhost", help='visdom server of the web display')
        parser.add_argument('--display_env', type=str, default='main', help='visdom display environment name (default is "main")')
        parser.add_argument('--display_port', type=int, default=8097, help='visdom port of the web display')
        parser.add_argument('--update_html_freq', type=int, default=1000, help='frequency of saving training results to html')
        parser.add_argument('--print_freq', type=int, default=100, help='frequency of showing training results on console')
        parser.add_argument('--no_html', action='store_true', help='do not save intermediate training results to [opt.checkpoints_dir]/[opt.name]/web/')
        # network saving and loading parameters
        parser.add_argument('--save_latest_freq', type=int, default=5000, help='frequency of saving the latest results')
        parser.add_argument('--save_epoch_freq', type=int, default=5, help='frequency of saving checkpoints at the end of epochs')
        parser.add_argument('--save_by_iter', action='store_true', help='whether saves model by iteration')
        parser.add_argument('--continue_train', action='store_true', help='continue training: load the latest model')
        parser.add_argument('--epoch_count', type=int, default=1, help='the starting epoch count, we save the model by <epoch_count>, <epoch_count>+<save_latest_freq>, ...')
        parser.add_argument('--phase', type=str, default='train', help='train, val, test, etc')
        # training parameters
        parser.add_argument('--n_epochs', type=int, default=100, help='number of epochs with the initial learning rate')
        parser.add_argument('--n_epochs_decay', type=int, default=100, help='number of epochs to linearly decay learning rate to zero')
        parser.add_argument('--beta1', type=float, default=0.5, help='momentum term of adam')
        parser.add_argument('--lr', type=float, default=0.0002, help='initial learning rate for adam')
        parser.add_argument('--gan_mode', type=str, default='lsgan', help='the type of GAN objective. [vanilla| lsgan | wgangp]. vanilla GAN loss is the cross-entropy objective used in the original GAN paper.')
        parser.add_argument('--pool_size', type=int, default=50, help='the size of image buffer that stores previously generated images')
        parser.add_argument('--lr_policy', type=str, default='linear', help='learning rate policy. [linear | step | plateau | cosine]')
        parser.add_argument('--lr_decay_iters', type=int, default=50, help='multiply by a gamma every lr_decay_iters iterations')

        self.isTrain = True
        return parser

Image Quality Assessment

使用IQA文件夹中的MATLAB代码，在MATLAB中运行main来对生成图像的质量进行评价主要关注PSNR、SSIM、FSIM 运行结果如下图：

生成图片：

用Resnet-18网络判别生成的图像是否是SAR图像

环境搭建

1、创建虚拟环境运行如下代码： conda create classification python=3.7

2、查看电脑CUDA版本 NVIDIA设置 -> 系统信息 -> 组件

3、根据CUDA版本安装pytorch 进入pytorch官网查看pytorch安装命令：pytorch官网 CUDA=10.2 conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

4、安装matplotlib python -m pip install -U matplotlib

数据集设置

训练集：

datasets---opt2sar---train---sar
                          ---opt
                  ----test---sar
                          ---opt
                  -----val---sar
                          ---opt

测试集：

data---datatest---test

运行

python resnet18.py 运行完成后生成resnet.pt文件运行完成图片： <img src=./images/resnet18train.png>

测试

python test.py

测试图片：

参考

CycleGAN源码：pytorch-CycleGAN-and-pix2pix

IQA代码来自：IQA

resnet18代码来自：resnet18增添了测试文件