Awesome

Unveiling Deep Shadows: A Survey on Image and Video Shadow Detection, Removal, and Generation in the Era of Deep Learning

by Xiaowei Hu*, Zhenghao Xing*, Tianyu Wang, Chi-Wing Fu, and Pheng-Ann Heng

https://arxiv.org/abs/2409.02108

中文解读1, 中文解读2

This repository contains the results and trained models for deep-learning methods used in shadow detection, removal, and generation, as presented in our paper "Unveiling Deep Shadows: A Survey on Image and Video Shadow Detection, Removal, and Generation in the Era of Deep Learning." This paper presents a comprehensive survey of shadow detection, removal, and generation in images and videos within the deep learning landscape over the past decade, covering tasks, deep models, datasets, and evaluation metrics. Key contributions include a comprehensive survey of shadow analysis, standardization of experimental comparisons, exploration of the relationships among model size, speed, and performance, a cross-dataset generalization study, identification of open issues and future directions, and provision of publicly available resources to support further research.

Highlights

• A Comprehensive Survey of Shadow Analysis in the Deep Learning Era.
• Fair Comparisons of the Existing Methods. Unified platform + newly refined datasets with corrected noisy labels and ground-truth images.
• Exploration of Model Size, Speed, and Performance Relationships. A more comprehensive comparison of different evaluation aspects.
• Cross-Dataset Generalization Study. Assess the generalization capability of deep models across diverse datasets.
• Overview of Open Issues and Future Directions with AIGC and Large Models.
• Publicly Available Results, Trained Models, and Evaluation Metrics.

Table of Contents

• Image Shadow Detection
  • Comparing image shadow detection methods on SBU-Refine and CUHK-Shadow
  • Cross-dataset generalization evaluation trained on SBU-Refine and tested on SRD
  • Qualitative Results
  • Weights and Code
  • Datasets
  • Metric
  • Papers
• Video Shadow Detection
  • Comparison of video shadow detection methods on ViSha
  • Qualitative Results
  • Weights and Code
  • Dataset
  • Metrics
  • Papers
• Instance Shadow Detection
  • Comparing image instance shadow detection methods on the SOBA-testing set
  • Comparing image instance shadow detection methods on the SOBA-challenge set
  • Cross-dataset generalization evaluation trained on SOBA and tested on SOBA-VID
  • Qualitative Results
  • Weights and Code
  • Datasets
  • Metrics
  • Papers
• Image Shadow Removal
  • Comparing image shadow removal methods on SRD and ISTD+
  • Cross-dataset generalization evaluation trained on SRD and tested on DESOBA
  • Qualitative Results
  • Weights and Code
  • Datasets
  • Metrics
  • Papers
• Video Shadow Removal
  • Papers
• Document Shadow Removal
  • Comparing document shadow removal methods on RDD
  • Qualitative Results
  • Weights and Code
  • Dataset
  • Papers
• Facial Shadow Removal
  • Papers
• Image Shadow Generation
  • Papers
• Bibtex

Image Shadow Detection

Comparing image shadow detection methods on SBU-Refine and CUHK-Shadow

Shadow detection methods on the SBU-Refine (left) and CUHK-Shadow (right) datasets: accuracy, parameters (indicated by the area of the bubbles), and speed.

Notes:

• Evaluation on an NVIDIA GeForce RTX 4090 GPU
• $: additional training data
• *: real-time shadow detector
• #: extra supervision from other methods

Cross-dataset generalization evaluation trained on SBU-Refine and tested on SRD

Qualitative Results

• Results for SBU-Refine and CUHK-Shadow Datasets:

  • Results-Part1
  • Results-Part2

• Cross-Dataset Generalization Evaluation:

  Trained on SBU-Refine and tested on SRD dataset:

  • Cross-Dataset Results

Weights and Code

You can download the weights and find the code repositories for these models below:

Datasets

• SBU-Refine
• CUHK-Shadow
• SRD Training, Test, and Masks

Metric

• BER: this repository

Papers

1. CNN-CRF:
   • Khan et al., "Automatic feature learning for robust shadow detection", CVPR, 2014. [paper]
   • Khan et al., "Automatic shadow detection and removal from a single image", TPAMI, 2016. [paper]
2. SCNN-LinearOpt: Shen et al., "Shadow optimization from structured deep edge detection", CVPR, 2015. [paper]
3. Stacked-CNN:
   • Vicente et al., "Large-scale training of shadow detectors with noisily-annotated shadow examples", ECCV, 2016. [paper]
   • Hou et al., "Large scale shadow annotation and detection using lazy annotation and stacked CNNs", TPAMI, 2021. [paper]
4. scGAN: Nguyen et al., "Shadow detection with conditional generative adversarial networks", ICCV, 2017. [paper]
5. Patched-CNN: Hosseinzadeh et al., "Fast shadow detection from a single image using a patched convolutional neural network", IROS, 2018. [paper]
6. ST-CGAN: Wang et al., "Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal", CVPR, 2018. [paper]
7. DSC:
   • Hu et al., "Direction-aware spatial context features for shadow detection", CVPR, 2018. [paper]
   • Hu et al., "Direction-aware spatial context features for shadow detection and removal", TPAMI, 2020. [paper]
8. DC-DSPF: Wang et al., "Densely cascaded shadow detection network via deeply supervised parallel fusion", IJCAI, 2018. [paper]
9. CPNet: Mohajerani et al., "CPNet: A context preserver convolutional neural network for detecting shadows in single RGB images", MMSP, 2018. [paper]
10. A+D Net: Le et al., "A+D Net: Training a shadow detector with adversarial shadow attenuation", ECCV, 2018. [paper]
11. BDRAR: Zhu et al., "Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection", ECCV, 2018. [paper]
12. DSDNet: Zheng et al., "Distraction-aware shadow detection", CVPR, 2019. [paper]
13. ARGAN/ARGAN+SS: Ding et al., "ARGAN: Attentive recurrent generative adversarial network for shadow detection and removal", ICCV, 2019. [paper]
14. CPAdv-Net: Mohajerani et al., "Shadow detection in single RGB images using a context preserver convolutional neural network trained by multiple adversarial examples", TIP, 2019. [paper]
15. MTMT-Net: Chen et al., "A multi-task mean teacher for semi-supervised shadow detection", CVPR, 2020. [paper]
16. DSSDNet: Luo et al., "Deeply supervised convolutional neural network for shadow detection based on a novel aerial shadow imagery dataset", P&RS, 2020. [paper]
17. FDRNet: Zhu et al., "Mitigating intensity bias in shadow detection via feature decomposition and reweighting", ICCV, 2021. [paper]
18. FSDNet: Hu et al., "Revisiting shadow detection: A new benchmark dataset for complex world", TIP, 2021. [paper]
19. ECA: Fang et al., "Robust shadow detection by exploring effective shadow contexts", ACM MM, 2021. [paper]
20. RCMPNet: Liao et al., "Shadow detection via predicting the confidence maps of shadow detection methods", ACM MM, 2021. [paper]
21. SDCM: Zhu et al., "Single image shadow detection via complementary mechanism", ACM MM, 2022. [paper]
22. TransShadow: Jie et al., "A fast and efficient network for single image shadow detection", ICASSP, 2022. [paper]
23. SILT: Yang et al., "SILT: Shadow-aware iterative label tuning for learning to detect shadows from noisy labels", ICCV, 2023. [paper]
24. Sun et al.: Sun et al., "Adaptive illumination mapping for shadow detection in raw images", ICCV, 2023. [paper]
25. R2D: Valanarasu et al., "Fine-context shadow detection using shadow removal", WACV, 2023. [paper]
26. LRA&LDRA: Yücel et al., "LRA&LDRA: Rethinking residual predictions for efficient shadow detection and removal", WACV, 2023. [paper]
27. SDDNet: Cong et al., "SDDNet: Style-guided dual-layer disentanglement network for shadow detection", ACM MM, 2023. [paper]
28. SDTR/SDTR+: Wu et al., "How many annotations do we need for generalizing new-coming shadow images?", TCSVT, 2023. [paper]
29. SAM-Adapter: Chen et al., "SAM-Adapter: Adapting segment anything in underperformed scenes", ICCVW, 2023. [paper]
30. ShadowSAM: Chen et al., "Make segment anything model perfect on shadow detection", TGRS, 2023. [paper]
31. AdapterShadow: Jie et al., "AdapterShadow: Adapting segment anything model for shadow detection", arXiv, 2023. [paper]

Video Shadow Detection

Comparison of video shadow detection methods on ViSha

Notes:

• Evaluation on an NVIDIA GeForce RTX 4090 GPU
• $: additional training data
• *: real-time shadow detector

Qualitative Results

• Results for ViSha Dataset:

  • Results

Weights and Code

You can download the weights and find the code repositories for these models below:

Dataset

• ViSha

Metrics

• TS: this repository & pre-computed optical flow
• IoU: this repository
• BER: this repository

Papers

1. TVSD-Net: Chen et al., "Triple-cooperative video shadow detection", CVPR, 2021. [paper]

2. Hu et al.: Hu et al., "Temporal feature warping for video shadow detection", arXiv, 2021. [paper]

3. STICT: Lu et al., "Video shadow detection via spatio-temporal interpolation consistency training", CVPR, 2022. [paper]

4. SC-Cor: Ding et al., "Learning shadow correspondence for video shadow detection", ECCV, 2022. [paper]

5. STF-Net: Lin and Wang, "Spatial-temporal fusion network for fast video shadow detection", VRCAI, 2022. [paper]

6. SCOTCH and SODA: Liu et al., "SCOTCH and SODA: a transformer video shadow detection framework", CVPR, 2023. [paper]

7. ShadowSAM: Wang et al., "Detect any shadow: segment anything for video shadow detection", TCSVT, 2023. [paper]

8. RSM-Net: Wang et al., "Language-driven interactive shadow detection", ACM MM, 2024. [paper]

9. TBGDiff: Zhou et al., "Timeline and boundary guided diffusion network for video shadow detection", ACM MM, 2024. [paper]

10. Duan et al.: Duan et al., "Two-stage video shadow detection via temporal-spatial adaption", ECCV, 2024. [paper]

Instance Shadow Detection

Comparing image instance shadow detection methods on the SOBA-testing set

Comparing image instance shadow detection methods on the SOBA-challenge set

Cross-dataset generalization evaluation trained on SOBA and tested on SOBA-VID

Qualitative Results

• Results for SOBA and SOBA-challenge Datasets:

  • Results-SOBA
  • Results-SBOA-challenge

• Cross-Dataset Generalization Evaluation:

  Trained on SOBA and tested on SOBA-VID dataset:

  • Cross-Dataset Results

Weights and Code

You can download the weights and find the code repositories for these models below:

Datasets

• SOBA
• SOBA-VID

Metrics

• SOAP: Python
• SOAP-VID: Python

Papers

1. LISA: Wang et al., "Instance shadow detection", CVPR, 2020. [paper]
2. SSIS: Wang et al., "Single-stage instance shadow detection with bidirectional relation learning", CVPR, 2021. [paper]
3. SSISv2: Wang et al., "Instance shadow detection with a single-stage detector", TPAMI, 2023. [paper]
4. ViShadow: Xing et al., "Video instance shadow detection under the sun and sky", TIP, 2024. [paper]

Image Shadow Removal

Shadow removal methods on the SRD (left) and ISTD+ (right) datasets: accuracy, parameters (indicated by the area of the bubbles), and speed.

Comparing image shadow removal methods on SRD and ISTD+

Notes:

• Evaluation on an NVIDIA GeForce RTX 4090 GPU
• LPIPS uses VGG as the extractor.
• Mask-ShadowGAN and DC-ShadowNet are unsupervised methods.
• G2R-ShadowNet is a weakly-supervised method.
• The PyTorch implementation of DSC for shadow removal based on DSC (Caffe).
• The unofficial PyTorch implementation of ST-CGAN.

Cross-dataset generalization evaluation trained on SRD and tested on DESOBA

Notes:

• DESOBA only labels cast shadows and we set the self shadows on objects as "don’t care" in evaluation.

Qualitative Results

• Results for SRD and ISTD+ Datasets:

  • Results-Part1
  • Results-Part2
  • Results-Part3

• Cross-Dataset Generalization Evaluation:

  Trained on SRD and tested on DESOBA dataset:

  • Results-Part1

  • Results-Part2

  • Results-Part3

  • Results-Part4

  • Results-Part5

Weights and Code

You can download the weights and find the code repositories for these models below:

Datasets

• SRD Training, Test, and Masks
• ISTD+
• DESOBA

Metrics

• RMSE: this repository
• PSNE: this repository
• SSIM: this repository
• LPIPS: this repository
• RMSE & PSNR with don't care: this repository

Papers

1. CNN-CRF: Khan et al., "Automatic shadow detection and removal from a single image", TPAMI, 2016. [paper]
2. DeshadowNet: Qu et al., "DeshadowNet: A multi-context embedding deep network for shadow removal", CVPR, 2017. [paper]
3. ST-CGAN: Wang et al., "Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal", CVPR, 2018. [paper]
4. AngularGAN: Sidorov et al., "Conditional GANs for multi-illuminant color constancy: revolution or yet another approach?", CVPRW, 2019. [paper]
5. ARGAN/ARGAN+SS: Ding et al., "ARGAN: Attentive recurrent generative adversarial network for shadow detection and removal", ICCV, 2019. [paper]
6. SP+M-Net: Le et al., "Shadow removal via shadow image decomposition", ICCV, 2019. [paper]
7. Mask-ShadowGAN: Hu et al., "Mask-ShadowGAN: Learning to remove shadows from unpaired data", ICCV, 2019. [paper]
8. DSC: Hu et al., "Direction-aware spatial context features for shadow detection and removal", TPAMI, 2020. [paper]
9. RIS-GAN: Zhang et al., "RIS-GAN: Explore residual and illumination with generative adversarial networks for shadow removal", AAAI, 2020. [paper]
10. DHAN+DA: Cun et al., "Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting GAN", AAAI, 2020. [paper]
11. Param+M+D-Net: Le et al., "From shadow segmentation to shadow removal", ECCV, 2020. [paper]
12. SP+M+I-Net: Le et al., "Physics-based shadow image decomposition for shadow removal", TPAMI, 2021. [paper]
13. Auto: Fu et al., "Auto-exposure fusion for single-image shadow removal", CVPR, 2021. [paper]
14. G2R-ShadowNet: Liu et al., "From shadow generation to shadow removal", CVPR, 2021. [paper]
15. PUL: Vasluianu et al., "Shadow removal with paired and unpaired learning", CVPRW, 2021. [paper]
16. CANet: Chen et al., "CANet: A context-aware network for shadow removal", ICCV, 2021. [paper]
17. DC-ShadowNet: Jin et al., "DC-ShadowNet: Single-image hard and soft shadow removal using unsupervised domain-classifier guided network", ICCV, 2021. [paper]
18. LG-ShadowNet: Liu et al., "Shadow removal by a lightness-guided network with training on unpaired data", TIP, 2021. [paper]
19. EMDNet: Zhu et al., "Efficient model-driven network for shadow removal", AAAI, 2022. [paper]
20. CRFormer: Wan et al., "CRFormer: A cross-region transformer for shadow removal", arXiv, 2022. [paper]
21. BMNet: Zhu et al., "Bijective mapping network for shadow removal", CVPR, 2022. [paper]
22. G2C-DeshadowNet: Gao et al., "Towards real-world shadow removal with a shadow simulation method and a two-stage framework", CVPRW, 2022. [paper]
23. SG-ShadowNet: Wan et al., "Style-guided shadow removal", ECCV, 2022. [paper]
24. CNSNet: Yu et al., "CNSNet: A cleanness-navigated-shadow network for shadow removal", ECCVW, 2022. [paper]
25. ShadowDiffusion(J): Jin et al., "ShadowDiffusion: Diffusion-based shadow removal using classifier-driven attention and structure preservation", arXiv, 2022.
26. ShadowDiffusion(G): Guo et al., "ShadowDiffusion: When degradation prior meets diffusion model for shadow removal", CVPR, 2023. [paper]
27. MStructNet: Liu et al., "Structure-informed shadow removal networks", TIP, 2023. [paper]
28. Self-ShadowGAN: Jiang et al., "Learning to remove shadows from a single image", IJCV, 2023. [paper]
29. ShadowFormer: Guo et al., "ShadowFormer: Global context helps shadow removal", AAAI, 2023. [paper]
30. SpA-Former: Liu et al., "SpA-Former: An effective and lightweight transformer for image shadow removal", IJCNN, 2023. [paper]
31. WSRD: Vasluianu et al., "WSRD: A novel benchmark for high-resolution image shadow removal", CVPRW, 2023. [paper]
32. TSRFormer: Chang et al., "TSRFormer: Transformer-based two-stage refinement for single image shadow removal", CVPRW, 2023. [paper]
33. PES: Cui et al., "Pyramid ensemble structure for high-resolution image shadow removal", CVPRW, 2023. [paper]
34. BCDiff: Guo et al., "Boundary-aware divide and conquer: A diffusion-based solution for unsupervised shadow removal", ICCV, 2023. [paper]
35. Inpaint4shadow: Li et al., "Leveraging inpainting for single-image shadow removal", ICCV, 2023. [paper]
36. LRA&LDRA: Yücel et al., "LRA&LDRA: Rethinking residual predictions for efficient shadow detection and removal", WACV, 2023. [paper]
37. SHARDS: Sen et al., "SHARDS: Efficient shadow removal using dual stage network for high-resolution images", WACV, 2023. [paper]
38. TBRNet: Liu et al., "A shadow imaging bilinear model and three-branch residual network for shadow removal", TNNLS, 2023. [paper]
39. PRNet: Wang et al., "Progressive recurrent network for shadow removal", CVIU, 2024. [paper]
40. DeS3: Jin et al., "DeS3: Adaptive attention-driven self and soft shadow removal using ViT similarity", AAAI, 2024. [paper]
41. Recasting: Liu et al., "Recasting regional lighting for shadow removal", AAAI, 2024. [paper]
42. LFG-Diffusion: Mei et al., "Latent feature-guided diffusion models for shadow removal", WACV, 2024. [paper]
43. ShadowMaskFormer: Li et al., "ShadowMaskFormer: Mask augmented patch embeddings for shadow removal", arXiv, 2024. [paper]
44. ShadowRefiner: Dong et al., "ShadowRefiner: Towards mask-free shadow removal via fast Fourier transformer", CVPRW, 2024. [paper]
45. HomoFormer: Xiao et al., "HomoFormer: Homogenized transformer for image shadow removal", CVPR, 2024. [paper]
46. SG-GAN+DBRM: Zeng et al., "Semantic-guided adversarial diffusion model for self-supervised shadow removal", arXiv, 2024. [paper]
47. Diff-Shadow: Luo et al., "Diff-Shadow: Global-guided diffusion model for shadow removal", arXiv, 2024. [paper]

Video Shadow Removal

Papers

1. PSTNet: Chen et al., "Learning physical-spatiotemporal features for video shadow removal,” TCSVT, 2024. [paper]

Document Shadow Removal

Comparing document shadow removal methods on RDD

Notes:

• Evaluation on an NVIDIA GeForce RTX 4090 GPU
• LPIPS uses VGG as the extractor.

Qualitative Results

• Results for RDD Dataset:

  • Results

Weights and Code

You can download the weights and find the code repositories for these models below:

Dataset

• RDD

Papers

1. BEDSR-Net: Lin et al., "BEDSR-Net: A deep shadow removal network from a single document image", CVPR, 2020. [paper]
2. BGShadowNet: Zhang et al., "Document image shadow removal guided by color-aware background", CVPR, 2023. [paper]
3. FSENet: Li et al., "High-resolution document shadow removal via a large-scale real-world dataset and a frequency-aware shadow erasing net", ICCV, 2023. [paper]

Facial Shadow Removal

Papers

1. Zhang et al.: Zhang et al., "Portrait shadow manipulation", SIGGRAPH, 2020. [paper]
2. He et al.: He et al., "Unsupervised portrait shadow removal via generative priors", ACM MM, 2021. [paper]
3. GS+C: Liu et al., "Blind removal of facial foreign shadows", BMVC, 2022. [paper]
4. Lyu et al.: Lyu et al., "Portrait eyeglasses and shadow removal by leveraging 3D synthetic data", CVPR, 2022. [paper]
5. GraphFFNet: Zhang et al., "Facial image shadow removal via graph-based feature fusion", CGF (PG), 2023. [paper]

Image Shadow Generation

Papers

1. ShadowGAN: Zhang et al., "ShadowGAN: Shadow synthesis for virtual objects with conditional adversarial networks", CVM, 2019. (for Composition) [paper]
2. Mask-ShadowGAN: Hu et al., "Mask-ShadowGAN: Learning to remove shadows from unpaired data", ICCV, 2019. (for Shadow Removal) [paper]
3. ARShadowGAN: Liu et al., "ARShadowGAN: Shadow generative adversarial network for augmented reality in single light scenes", CVPR, 2020. (for Composition) [paper]
4. Zheng et al.: Zheng et al., "Learning to shadow hand-drawn sketches", CVPR, 2020. (for Sketch) [paper]
5. Shadow Matting GAN: Cun et al., "Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting GAN", AAAI, 2020. (for Shadow Removal) [paper]
6. SSN: Sheng et al., "SSN: Soft shadow network for image compositing", CVPR, 2021. (for Composition) [paper]
7. G2R-ShadowNet: Liu et al., "From shadow generation to shadow removal", CVPR, 2021. (for Shadow Removal) [paper]
8. SmartShadow: Zhang et al., "SmartShadow: Artistic shadow drawing tool for line drawings", ICCV, 2021. (for Sketch) [paper]
9. SSG: Sheng et al., "Controllable shadow generation using pixel height maps", ECCV, 2022. (for Composition) [paper]
10. SGRNet: Hong et al., "Shadow generation for composite image in real-world scenes", AAAI, 2022. (for Composition) [paper]
11. Liu et al.: Liu et al., "Shadow generation for composite image with multi-level feature fusion", EITCE, 2022. (for Composition) [paper]
12. PixHt-Lab: Sheng et al., "PixHt-Lab: Pixel height based light effect generation for image compositing", CVPR, 2023. (for Composition) [paper]
13. HAU-Net & IFNet: Meng et al., "Automatic shadow generation via exposure fusion", TMM, 2023. (for Composition) [paper]
14. Valença et al.: Valença et al., "Shadow harmonization for realistic compositing", SIGGRAPH Asia, 2023. (for Composition) [paper]
15. DMASNet: Tao et al., "DMASNet: Shadow generation with decomposed mask prediction and attentive shadow filling", AAAI, 2024. (for Composition) [paper]
16. SGDiffusion: Liu et al., "SGDiffusion: Shadow generation for composite image using diffusion model", CVPR, 2024. (for Composition) [paper]

Bibtex

If you find our work, results, models, and unified evaluation code useful, please consider citing our paper as follows:

@article{hu2024unveiling,
  title={Unveiling Deep Shadows: A Survey on Image and Video Shadow Detection, Removal, and Generation in the Era of Deep Learning},
  author={Hu, Xiaowei and Xing, Zhenghao and Wang, Tianyu and Fu, Chi-Wing and Heng, Pheng-Ann},
  journal={arXiv preprint arXiv:2409.02108},
  year={2024}
}