Home

Awesome

HGGT

Official PyTorch code and dataset for our paper "HGGT" in CVPR 2023.

Paper | Supplementary | Arxiv Version

Human Guided Ground-truth Generation for Realistic Image Super-resolution <br> Du CHEN*, Jie LIANG*, Xindong ZHANG, Ming LIU, Hui ZENG and Lei ZHANG. <br> Accepted by CVPR 2023.<br>

Copyright, License and Agreement for the HGGT dataset Usage

  1. Please note that this dataset is made available for non-commercial academic research purposes ONLY.
  2. You agree not to reproduce, duplicate, copy, sell, trade, resell or exploit for any commercial purposes, any portion of the images and any portion of derived data.
  3. You agree not to further copy, publish or distribute any portion of the HGGT dataset. Except, for internal use at a single site within the same organization it is allowed to make copies of the dataset.
  4. The image contents are released upon request for research purposes ONLY.
  5. Any violation of this protocol will be at his own risk. If any of the images include your information and you would like to remove them, please kindly inform us, and we will remove them from our dataset immediately.

Abstract

How to generate the ground-truth (GT) image is a critical issue for training realistic image super-resolution (Real- ISR) models. Existing methods mostly take a set of highresolution (HR) images as GTs and apply various degradations to simulate the low-resolution (LR) counterparts. Though great progress has been achieved, such an LR-HR pair generation scheme has several limitations. First, the perceptual quality of HR images may not be high enough, limiting the quality of Real-ISR model output. Second, existing schemes do not consider much human perception in GT generation, and the trained models tend to produce oversmoothed results or unpleasant artifacts. With the above considerations, we propose a human guided GT generation scheme. We first elaborately train multiple image enhancement models to improve the perceptual quality of HR images, and enable one LR image having multiple HR counterparts. Human subjects are then involved to annotate the high quality regions among the enhanced HR images as GTs, and label the regions with unpleasant artifacts as negative samples. A human guided GT image dataset with both positive and negative samples is then constructed, and a loss function is proposed to train the Real-ISR models. Experiments show that the Real-ISR models trained on our dataset can produce perceptually more realistic results with less artifacts.

Overall illustration of our dataset generation:

sketch map

For more details, please refer to our paper.

Getting started

git clone https://github.com/ChrisDud0257/HGGT
cd HGGT

Step 1: Image enhancement (Optional, you could just skip to step4 and train models with our proposed HGGT dataset)

Installation. (Python 3 + NVIDIA GPU + CUDA. Recommend to use Anaconda)

cd ImageEnhancement
conda create --name enhance python=3.9
pip install -r requirements.txt
python setup.py develop

We build up our project based on the widely-used BasicSR codeframe. For more installation details, please refer to the BasicSR framework.

Training

CUDA_VISIBLE_DEVICES=0 \
python ./basicsr/train.py -opt ./options/train/EnhancementModels/train_EnhanceRCAN_01.yml --auto_resume
cd ImageEnhancement
sh demo.sh

Inference

CUDA_VISIBLE_DEVICES=0 \
python ./inference/inference_rcannoup.py --input /home/chendu/data2_hdd10t/chendu/dataset/ForMultiGT/2/MultiGT1600 \
--output ./results/EnhancedGT/01 --model_path ./experiments/pretrained_models/enhancement/RCAN_01.pth --tile_size 1200 --suffix _01
cd ImageEnhancement
sh demo.sh
-EnhancedGT/
    -original/
        -img1.png
    -01/
        -img1_01.png
    -02/
        -img1_02.png
    -03/
        -img1_03.png
    -04/
        -img1_04.png

Step 2: Patch selection (Optional)

We complete Step 2 and Step 3 on Windows Operating System.

cd PatchSelection
python patch_select.py --img_path 'Path to EnhancedGT' --save_path 'Path to your own patch selection save path'
-PatchSelection/
    -original/
        -img1-x-y.png
        -img1-x1-y1.png
    -01/
        -img1-x-y_01.png
        -img1-x1-y1_01.png
    -02/
        -img1-x-y_02.png
        -img1-x1-y1_02.png
    -03/
        -img1-x-y_03.png
        -img1-x1-y1_03.png
    -04/
        -img1-x-y_04.png
        -img1-x1-y1_04.png
cd PatchSelection
python regroup.py --img_path 'Path to the patch selection save path' --save_path 'Path to your own regroup save path'

Step 3: Annotation (Optional)

We will provide the annotation software as well as the software tutorial in the future.

Ours HGGT dataset Structure

After the annotation step, we integrate the images and labels into the following folder structure:

Train/
    images/
        0-20193/
            original/
                img0-x-y.png
                img1-x1-y1.png
            01/
                img0-x-y_01.png
                img1-x1-y1_01.png
            02/
                img0-x-y_02.png
                img1-x1-y1_02.png
            03/
                img0-x-y_03.png
                img1-x1-y1_03.png
            04/
                img0-x-y_04.png
                img1-x1-y1_04.png
    labels/
        0-20193/
            A/
                img0-x-y.json
                img1-x1-y1.json
            B/
                img0-x-y.json
                img1-x1-y1.json
            C/
                img0-x-y.json
                img1-x1-y1.json
Picture_1 {'Name': '0001-586-1469.png', 'File': 'original', 'Label': None}
Picture_2 {'Name': '0001-586-1469_01.png', 'File': '01', 'Label': 'Positive'}
Picture_3 {'Name': '0001-586-1469_02.png', 'File': '02', 'Label': 'Positive'}
Picture_4 {'Name': '0001-586-1469_03.png', 'File': '03', 'Label': 'Positive'}
Picture_5 {'Name': '0001-586-1469_04.png', 'File': '04', 'Label': 'Positive'}
Time_cost {'Start': '2022-08-05 16:56:06', 'End': '2022-08-05 16:56:49', 'Total': 42}

Step 4: Train realistic Super-resolution models on our proposed HGGT dataset

Installation. (Python 3 + NVIDIA GPU + CUDA. Recommend to use Anaconda)

cd PosNegGTSR
conda create --name posnegsr python=3.9
pip install -r requirements.txt
python setup.py develop

We build up our project based on the widely-used BasicSR codeframe. For more installation details, please refer to the BasicSR framework.

Pre-train with out GAN (Optional)

CUDA_VISIBLE_DEVICES=0 \
python ./basicsr/train.py -opt ./options/train/PosNegGTSR/train_RRDB_DF2K_OST_Blind_x4.yml --auto_resume
cd PosNegGTSR
sh demo.sh

Training with a GAN with the original GT in our HGGT dataset or with DF2K_OST dataset.

CUDA_VISIBLE_DEVICES=0 \
python ./basicsr/train.py -opt ./options/train/PosNegGTSR/train_RRDBGAN_PosNegGT_Blind_Ori_x4.yml --auto_resume
cd PosNegGTSR
sh demo.sh
CUDA_VISIBLE_DEVICES=0 \
python ./basicsr/train.py -opt ./options/train/PosNegGTSR/train_LDL_DF2K_OST_Blind_x4.yml --auto_resume
cd PosNegGTSR
sh demo.sh

Trainging with the positive GTs in our HGGT dataset.

CUDA_VISIBLE_DEVICES=0 \
python ./basicsr/train.py -opt ./options/train/PosNegGTSR/train_RRDBGAN_PosNegGT_Blind_Pos_x4.yml --auto_resume
cd PosNegGTSR
sh demo.sh

Trainging with both positive and negative GTs in our HGGT dataset.

CUDA_VISIBLE_DEVICES=0 \
python ./basicsr/train.py -opt ./options/train/PosNegGTSR/train_RRDBGAN_PosNegGT_Blind_Pos+Neg_x4.yml --auto_resume
cd PosNegGTSR
sh demo.sh

Step 5: Testing with the models trained in step 4

To download our pre-trained models and for usage, please follow this instruction.

Testing with the model trained on the original GT in our proposed HGGT dataset or DF2K_OST dataset.

CUDA_VISIBLE_DEVICES=0 \
python ./basicsr/test.py -opt ./options/test/PosNegGTSR/test_RRDBGAN_PosNegGT_Blind_Ori_x4.yml
cd PosNegGTSR
sh demo.sh
CUDA_VISIBLE_DEVICES=0 \
python ./basicsr/test.py -opt ./options/test/PosNegGTSR/test_LDL_DF2K_OST_Blind_x4.yml
cd PosNegGTSR
sh demo.sh
CUDA_VISIBLE_DEVICES=0 \
python ./basicsr/test.py -opt ./options/test/PosNegGTSR/test_RRDBGAN_PosNegGT_Blind_Pos_x4.yml
cd PosNegGTSR
sh demo.sh
CUDA_VISIBLE_DEVICES=0 \
python ./basicsr/test.py -opt ./options/test/PosNegGTSR/test_RRDBGAN_PosNegGT_Blind_Pos+Neg_x4.yml
cd PosNegGTSR
sh demo.sh

Step 6: Evaluation metrics

PSNR & SSIM & LPIPS & DISTS

python ./scripts/metrics/calculate_multigt_labeled_psnr_ssim.py
python ./scripts/metrics/calculate_multigt_labeled_lpips.py
python ./scripts/metrics/calculate_multigt_labeled_dists.py
cd PosNegGTSR
sh demo.sh

SR results

We also provide all of the visualization and quantitative metrics results reported in our paper, you could download through GoogleDrive.

License

This project is released under the Apache 2.0 license.

Citation

@InProceedings{Chen_2023_CVPR,
    author    = {Chen, Du and Liang, Jie and Zhang, Xindong and Liu, Ming and Zeng, Hui and Zhang, Lei},
    title     = {Human Guided Ground-Truth Generation for Realistic Image Super-Resolution},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {14082-14091}
}

Acknowledgement

This project is built mainly based on the excellent BasicSR and KAIR codeframe. We appreciate it a lot for their developers.

Contact

If you have any questions or suggestions about this project, please contact me at csdud.chen@connect.polyu.hk .