Awesome

TBRNet

1. Resources

1.1 Dataset

1.2 Results

Results on SRD: TBRNet_SRD
Results on SRD+DA: TBRNet_SRD_DA
Results on ISTD: TBRNet_ISTD
Results on ISTD+DA: TBRNet_ISTD_DA
Results on ISTD+: TBRNet_ISTD+

Visual comparison results of penumbra removal results on the SRD dataset - (Powered by MulimgViewer) <img src="https://github.com/nachifur/TBRNet/blob/main/img/f1.jpg"/>

1.3 Model Weight File for Test

Model on SRD: TBRNet_SRD.pth
Model on SRD+DA: TBRNet_SRD_DA.pth
Model on ISTD: TBRNet_ISTD.pth
Model on ISTD+DA: TBRNet_ISTD_DA.pth
Model on ISTD+: TBRNet_ISTD+.pth

1.4 Evaluation Code

MAE (i.e., RMSE in paper). Hieu Le reuploaded the evaluation code. Before that, everyone can obtain this code from Jifeng Wang via email. Currently, the MATLAB code is used in most state-of-the-art works for shadow removal.
PSNR+SSIM. zhu_aaai2022

2. Environments

ubuntu18.04+cuda10.2+pytorch1.7.1

create environments

conda env create -f install.yaml

activate environments

conda activate TBRNet

3. Data Processing

For example, generate the dataset list of ISTD:

Download:

ISTD and SRD
USR shadowfree images
Syn. Shadow
SRD shadow mask (or try our re-uploaded link)
train_B_ISTD:

cp -r ISTD_Dataset_arg/train_B ISTD_Dataset_arg/train_B_ISTD
cp -r ISTD_Dataset_arg/train_B SRD_Dataset_arg/train_B_ISTD

VGG19

cp vgg19-dcbb9e9d.pth ISTD_Dataset_arg/
cp vgg19-dcbb9e9d.pth SRD_Dataset_arg/

The data folders should be:

ISTD_Dataset_arg
    * train
        - train_A # ISTD shadow image
        - train_B # ISTD shadow mask
        - train_C # ISTD shadowfree image
        - shadow_free # USR shadowfree images
        - synC # Syn. shadow
        - train_B_ISTD # ISTD shadow mask
    * test
        - test_A # ISTD shadow image
        - test_B # ISTD shadow mask
        - test_C # ISTD shadowfree image
    * vgg19-dcbb9e9d.pth

SRD_Dataset_arg
    * train #  renaming the original `Train` folder in `SRD`.
        - train_A # SRD shadow image, renaming the original `shadow` folder in `SRD`.
        - train_B # SRD shadow mask
        - train_C # SRD shadowfree image, renaming the original `shadow_free` folder in `SRD`.
        - shadow_free # USR shadowfree images
        - synC # Syn. shadow
        - train_B_ISTD # ISTD shadow mask
    * test #  renaming the original `test_data` folder in `SRD`.
        - train_A # SRD shadow image, renaming the original `shadow` folder in `SRD`.
        - train_B # SRD shadow mask
        - train_C # SRD shadowfree image, renaming the original `shadow_free` folder in `SRD`.
    * vgg19-dcbb9e9d.pth

Edit generate_flist_istd.py: (Replace path)

ISTD_path = "/Your_data_storage_path/ISTD_Dataset_arg"

Generate Datasets List. (Already contains ISTD+DA.)

conda activate TBRNet
cd script/
python generate_flist_istd.py

Edit config_ISTD.yml: (Replace path)

DATA_ROOT: /Your_data_storage_path/ISTD_Dataset_arg

4. Training+Test+Evaluation

4.1 Training+Test+Evaluation

For example, training+test+evaluation on ISTD dataset.

cp config/config_ISTD.yml config.yml 
cp config/run_ISTD.py run.py
conda activate TBRNet
python run.py

4.2 Only Test and Evaluation

For example, test+evaluation on ISTD dataset.

Download weight file(TBRNet_ISTD.pth) to pre_train_model/ISTD
Copy file

cp config/config_ISTD.yml config.yml 
cp config/run_ISTD.py run.py
mkdir -p checkpoints/ISTD/
cp config.yml checkpoints/ISTD/config.yml
cp pre_train_model/ISTD/TBRNet_ISTD.pth  checkpoints/ISTD/ShadowRemoval.pth

Edit run.py. Comment the training code.

    # # pre_train (no data augmentation)
    # MODE = 0
    # print('\nmode-'+str(MODE)+': start pre_training(data augmentation)...\n')
    # for i in range(1):
    #     skip_train = init_config(checkpoints_path, MODE=MODE,
    #                             EVAL_INTERVAL_EPOCH=1, EPOCH=[90,i])
    #     if not skip_train:
    #         main(MODE, config_path)
    # src_path = Path('./pre_train_model') / \
    #     config["SUBJECT_WORD"]/(config["MODEL_NAME"]+'_pre_da.pth')
    # copypth(dest_path, src_path)

    # # train
    # MODE = 2
    # print('\nmode-'+str(MODE)+': start training...\n')
    # for i in range(1):
    #     skip_train = init_config(checkpoints_path, MODE=MODE,
    #                             EVAL_INTERVAL_EPOCH=0.1, EPOCH=[60,i])
    #     if not skip_train:
    #         main(MODE, config_path)
    # src_path = Path('./pre_train_model') / \
    #     config["SUBJECT_WORD"]/(config["MODEL_NAME"]+'_final.pth')
    # copypth(dest_path, src_path)

conda activate TBRNet
python run.py

4.3 Show Results

After evaluation, execute the following code to display the final RMSE.

python show_eval_result.py

Output:

running rmse-shadow: xxx, rmse-non-shadow: xxx, rmse-all: xxx # ISRD

This is the evaluation result of python+pytorch, which is only used during training. To get the evaluation results in the paper, you need to run the matlab code.

5. Acknowledgements

Part of the code is based upon:

6. Citation

If you find our work useful in your research, please consider citing:

@ARTICLE{liu2023shadow,
  author={Liu, Jiawei and Wang, Qiang and Fan, Huijie and Tian, Jiandong and Tang, Yandong},
  journal={IEEE Transactions on Neural Networks and Learning Systems}, 
  title={A Shadow Imaging Bilinear Model and Three-Branch Residual Network for Shadow Removal}, 
  year={2023},
  volume={},
  number={},
  pages={1-15},
  doi={10.1109/TNNLS.2023.3290078}
}

7. Contact

Please contact Jiawei Liu if there is any question (liujiawei18@mails.ucas.ac.cn).