Awesome

Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising

Miaoyu Li, Ji Liu, Ying Fu, Yulun Zhang, and Dejing Dou, "Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising" CVPR 2023

Arxiv

Abstract: Denoising is a crucial step for Hyperspectral image (HSI) applications. Though witnessing the great power of deep learning, existing HSI denoising methods suffer from limitations in capturing the non-local self-similarity. Transformers have shown potential in capturing long-range dependencies, but few attempts have been made with specifically designed Transformer to model the spatial and spectral correlation in HSIs. In this paper, we address these issues by proposing a spectral enhanced rectangle Transformer, driving it to explore the non-local spatial similarity and global spectral low-rank property of HSIs. For the former, we exploit the rectangle self-attention horizontally and vertically to capture the non-local similarity in the spatial domain. For the latter, we design a spectral enhancement module that is capable of extracting global underlying low-rank property of spatial-spectral cubes to suppress noise, while enabling the interactions among non-overlapping spatial rectangles. Extensive experiments have been conducted on both synthetic noisy HSIs and real noisy HSIs, showing the effectiveness of our proposed method in terms of both objective metric and subjective visual quality.

Network Architecture

Contents

1. Models
2. Datasets
3. Training and Testing
4. Results

<a id="Models"></a>

Models

We also provide the checkpoints of our competing methods.

Quantitative Comparison on ICVL Dataset

Quantitative Comparison on Realistic Dataset

Quantitative Comparison on Apex/Urban Dataset

<a id="Datasets"></a>

Datasets

ICVL

• The entire ICVL dataset download link: https://icvl.cs.bgu.ac.il/hyperspectral/

1. split the entire dataset into training samples, testing samples and validating samples. The files used in training are listed in utility/icvl_partition/icvl_train_list.txt.
2. generate lmdb dataset for training

python utility/lmdb_data.py

3. download the testing data from BaiduDisk or generate them by yourself through

python utility/mat_data.py

Realistic Dataset

• Please refer to [github-link] for "Hyperspectral Image Denoising with Realistic Data in ICCV, 2021" to download the dataset

Testing HSIs are listed in utility/realistic_partition/test.txt. Other HSIs in the dataset are used for training.

Urban dataset

• The training dataset are from link: https://apex-esa.org/. The origin Urban dataset are from link: https://rslab.ut.ac.ir/data.

1. Run the create_big_apex_dataset() funtion in utility/mat_data.py to generate training samples.

2. Run the createDCmall() function in utility/lmdb_data.py to generate training lmdb dataset.

<a id="Training"></a>

Training and Testing

ICVL Dataset

#for gaussian noise
#----training----
python hside_simu.py -a sert_base -p sert_base_gaussian

#----testing---- The results are shown in Table 1 in the main paper.
python hside_simu_test.py -a sert_base -p sert_base_gaussian_test -r -rp checkpoints/icvl_gaussian.pth --test-dir /icvl_noise/512_50

#for comlpex noise
#----training----
python hside_simu_complex.py -a sert_base -p sert_base_complex

#----testing---- The results are shown in Table 2 in the main paper.
python hside_simu_test.py -a sert_base -p sert_base_complex_test -r -rp checkpoints/icvl_complex.pth --test-dir  /icvl_noise/512_mix

Urban Dataset

#----training----
python hside_urban.py -a sert_urban -p sert_urban 

#----testing----  The results are shown in Figure 4 in the main paper.
python hside_urban_test.py -a sert_urban -p sert_urban_test -r -rp ./checkpoints/real_urban.pth

Realistic Dataset

#----training----
python hside_real.py -a sert_real -p sert_real

#----testing---- The results are shown in Table 3 in the main paper.
python hside_real_test.py -a sert_real -p sert_real_test -r -rp ./checkpoints/real_realistic.pth

<a id="Results"></a>

Results

Citations

If you find the code helpful in your resarch or work, please cite the following paper(s).

@inproceedings{li2023spatial,
    title={Spatial-Spectral Transformer for Hyperspectral Image Denoising},
    author={Li, Miaoyu and Fu, Ying and Zhang, Yulun},
    booktitle={AAAI},
    year={2023}
}
@inproceedings{li2023spectral,
    title={Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising},
    author={Miaoyu Li and Ji Liu and Ying Fu and Yulun Zhang and Dejing Dou},
    booktitle={CVPR},
    year={2023}
}

Acknowledgement

The codes are based on QRNN3D.