Awesome

Title: Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey

Description: This project is part of our survey. The survey paper can be downloaded on： https://arxiv.org/abs/2405.01725

Part I: Residual_Learning_In_Deep_Neural_Networks

1. Short and long skip connection

Related papers: <br> (1) Fully convolutional networks for semantic segmentation

• Paper: https://arxiv.org/abs/1411.4038

(2) U-Net: Convolutional Networks for Biomedical Image Segmentation

• Paper: https://arxiv.org/abs/1505.04597
• Code:https://github.com/tsmanral/U-Net-Convolutional-Networks-for-Biomedical-Image-Segmentation

(3) SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling

• Paper: https://arxiv.org/abs/1505.07293
• Code: https://github.com/preddy5/segnet

(4) DiSegNet: A deep dilated convolutional encoder-decoder architecture for lymph node segmentation on PET/CT images

• Paper: https://www.sciencedirect.com/science/article/abs/pii/S0895611120301464

(5) UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation

• Paper: https://arxiv.org/abs/1912.05074
• Code: https://github.com/MrGiovanni/UNetPlusPlus

(6) nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation

• Paper: https://www.nature.com/articles/s41592-020-01008-z
• Code: https://github.com/MIC-DKFZ/nnUNet

(7) Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections

• Paper: https://arxiv.org/abs/1603.09056

(8) Densely Connected Convolutional Networks

• Paper: https://arxiv.org/abs/1608.06993
• Code: https://github.com/liuzhuang13/DenseNet

(9) Rapid whole-brain electric field mapping in transcranial magnetic stimulation using deep learning

• Paper: https://pubmed.ncbi.nlm.nih.gov/34329328/

(10) ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data

• Paper: https://www.sciencedirect.com/science/article/abs/pii/S0924271620300149
• Code: https://github.com/feevos/resuneta

(11) Volumetric ConvNets with mixed residual connections for automated prostate segmentation from 3D MR images

• Paper: https://www.researchgate.net/publication/312232002_Volumetric_ConvNets_with_Mixed_Residual_Connections_for_Automated_Prostate_Segmentation_from_3D_MR_Images

2. Widen the residual block

Related papers: <br> (1) Going Deeper with Convolutions

• Paper: https://arxiv.org/abs/1409.4842

(2) Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

• Paper: https://arxiv.org/abs/1602.07261

(3) Aggregated Residual Transformations for Deep Neural Networks

• Paper: https://arxiv.org/abs/1611.05431
• Code: https://github.com/facebookresearch/ResNeXt

(4) Res2Net: A New Multi-scale Backbone Architecture

• Paper: https://arxiv.org/abs/1904.01169
• Code: https://mmcheng.net/res2net/

(5) Wide Residual Networks

• Paper: https://arxiv.org/abs/1605.07146
• Code: https://github.com/szagoruyko/wide-residual-networks

3. Make effective residual block

Related papers:<br> (1) Identity Mappings in Deep Residual Networks

• Paper: https://arxiv.org/abs/1603.05027
• Code: https://github.com/KaimingHe/resnet-1k-layers

(2) Squeeze-and-excitation networks

• Paper: https://arxiv.org/abs/1709.01507
• Code: https://github.com/hujie-frank/SENet

(3) Selective kernel networks

• Paper: https://arxiv.org/abs/1903.06586
• Code: https://github.com/implus/SKNet

(4) Resnest: Split-attention networks

• Paper: https://arxiv.org/abs/2004.08955
• Code: https://github.com/zhanghang1989/ResNeSt

(5) Cbam: Convolutional block attention module

• Paper: https://arxiv.org/abs/1807.06521

(6) Dual Attention Network for Scene Segmentation

• Paper: https://arxiv.org/abs/1809.02983
• Code: https://github.com/niecongchong/DANet-keras

(7) Mobilenetv2: Inverted residuals and linear bottlenecks

• Paper: https://arxiv.org/abs/1801.04381

(8) Rethinking bottleneck structure for efficient mobile network design

• Paper: https://arxiv.org/abs/2007.02269
• Code: https://github.com/zhoudaquan/rethinking_bottleneck_design

(9) Ghostnet: More features from cheap operations

• Paper: https://arxiv.org/abs/1911.11907
• Code: https://github.com/huawei-noah/Efficient-AI-Backbones

(10) Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

• Paper: https://arxiv.org/abs/1412.7062

(11) DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

• Paper: https://arxiv.org/abs/1606.00915

(12) Rethinking atrous convolution for semantic image segmentation

• Paper: https://arxiv.org/abs/1706.05587

(13) Encoder-decoder with atrous separable convolution for semantic image segmentation

• Paper: https://arxiv.org/abs/1802.02611
• Code: https://github.com/tensorflow/models/tree/master/research/deeplab

(14) StereoDRNet: Dilated Residual Stereo Net

• Paper: https://arxiv.org/abs/1904.02251

(15) Deep Pyramidal Residual Networks

• Paper: https://arxiv.org/abs/1610.02915
• Code: https://github.com/jhkim89/PyramidNet

(16) Compressed Sensing MRI via a Multi-scale Dilated Residual Convolution Network

• Paper: https://arxiv.org/abs/1906.05251

(17) A Multiscale Image Denoising Algorithm Based On Dilated Residual Convolution Network

• Paper: https://arxiv.org/abs/1812.09131

(18) Multi-level dilated residual network for biomedical image segmentation

• Paper: https://pubmed.ncbi.nlm.nih.gov/34238940

(19) ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation

• Paper: https://arxiv.org/abs/1803.06815
• Code: https://github.com/sacmehta/ESPNet

(20) ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network

• Paper: https://arxiv.org/abs/1811.11431
• Code: https://github.com/sacmehta/ESPNetv2

(21) Deformable Convolutional Networks

• Paper: https://arxiv.org/abs/1703.06211
• Code: https://github.com/msracver/Deformable-ConvNets

(22)Temporal deformable residual networks for action segmentation in videos

• Paper: https://www.researchgate.net/publication/324794749_Temporal_Deformable_Residual_Networks_for_Action_Segmentation_in_Videos

(23) Deformable and residual convolutional network for image super-resolution

• Paper: https://www.researchgate.net/publication/351126732_Deformable_and_residual_convolutional_network_for_image_super-resolution

(24) Deformable 3D Convolution for Video Super-Resolution

• Paper: https://arxiv.org/abs/2004.02803
• Code: https://github.com/XinyiYing/D3Dnet

(25) A Spectral Spatial Attention Fusion with Deformable Convolutional Residual Network for Hyperspectral Image Classification

• Paper: https://www.mdpi.com/2072-4292/13/18/3590

(26) DCU-net: a deformable convolutional neural network based on cascade U-net for retinal vessel segmentation

• Paper: https://link.springer.com/article/10.1007/s11042-022-12418-w

4. Design efficient residual-based architectures

Related papers: <br> (1)Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications

• Paper: https://arxiv.org/abs/2204.11786

(2) Network in network

• Paper: https://arxiv.org/abs/1312.4400

(3) Aggregated Residual Transformations for Deep Neural Networks

• Paper: https://arxiv.org/abs/1611.05431
• Code: https://github.com/facebookresearch/ResNeXt

(4) ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

• Paper: https://arxiv.org/abs/1707.01083

(5) MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

• Paper: https://arxiv.org/abs/1704.04861

(6) ImageNet classification with deep convolutional neural networks

• Paper: https://dl.acm.org/doi/10.1145/3065386

(7) Xception: Deep Learning with Depthwise Separable Convolutions

• Paper: https://arxiv.org/abs/1610.02357

(8) Searching for MobileNetV3

• Paper: https://arxiv.org/abs/1905.02244
• Code: https://github.com/kuan-wang/pytorch-mobilenet-v3

(9) Deep Networks with Stochastic Depth

• Paper: https://arxiv.org/abs/1603.09382

(10) Neural Network Pruning with Residual-Connections and Limited-Data

• Paper: https://arxiv.org/abs/1911.08114

(11) ResKD: Residual-Guided Knowledge Distillation

• Paper: https://arxiv.org/abs/2006.04719

5. Self-attention in residual learning

Related papers: <br> (1) Non-local Neural Networks

• Paper: https://arxiv.org/abs/1711.07971
• Code: https://github.com/facebookresearch/video-nonlocal-net

(2) Ccnet:Criss-cross attention for semantic segmentation

• Paper: https://arxiv.org/abs/1811.11721
• Code: https://github.com/speedinghzl/CCNet

(3) Expectation-Maximization Attention Networks for Semantic Segmentation

• Paper: https://arxiv.org/abs/1907.13426
• Code: https://github.com/XiaLiPKU/EMANet

(4) An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

• Paper: https://arxiv.org/abs/2010.11929
• Code: https://github.com/google-research/vision_transformer

(5) Swin transformer: Hierarchical vision transformer using shifted windows

• Paper: https://arxiv.org/abs/2103.14030
• Code: https://github.com/microsoft/Swin-Transformer

(6) End-to-End Object Detection with Transformers

• Paper: https://arxiv.org/abs/2005.12872
• Code: https://github.com/facebookresearch/detr

(7) Deformable DETR: Deformable Transformers for End-to-End Object Detection

• Paper: https://arxiv.org/abs/2010.04159
• Code: https://github.com/fundamentalvision/Deformable-DETR

(8) Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

• Paper: https://arxiv.org/abs/2012.15840
• Code: https://fudan-zvg.github.io/SETR/

(9) TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation

• Paper: https://arxiv.org/abs/2102.04306
• Code: https://github.com/Beckschen/TransUNet

(10) TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up

• Paper: https://arxiv.org/abs/2102.07074
• Code: https://github.com/VITA-Group/TransGAN

(11) Zero-Shot Text-to-Image Generation

• Paper: https://arxiv.org/abs/2102.12092

Part II: Residual learning in image classification, object detection, semantic segmentation, image compression, deblur, denoise and Super-Resolution

1. Deep neural networks based on residual learning for image classification

(1) Deep Residual Learning for Image Recognition

• Paper: https://arxiv.org/abs/2312.00109
• Code: https://github.com/city-super/Scaffold-GS](https://github.com/KaimingHe/deep-residual-networks

(2) Highway Networks

• Paper: https://arxiv.org/abs/1505.00387
• Code: https://github.com/c0nn3r/pytorch_highway_networks

(3)Deep Pyramidal Residual Networks

• Paper: https://arxiv.org/abs/1610.02915
• Code: https://github.com/dyhan0920/PyramidNet-PyTorch

(4)Wider or Deeper: Revisiting the ResNet Model for Visual Recognition

• Paper: https://arxiv.org/abs/1611.10080
• Code: https://github.com/itijyou/ademxapp

(5)Aggregated Residual Transformations for Deep Neural Networks

• Paper: https://arxiv.org/abs/1611.05431
• Code: https://github.com/prlz77/ResNeXt.pytorch

(6)Densely Connected Convolutional Networks

• Paper: https://arxiv.org/abs/1608.06993
• Code: https://github.com/bamos/densenet.pytorch

(7)DPN

• Code: https://github.com/huggingface/pytorch-image-models?tab=readme-ov-file#citing

(8)Invertible Residual Networks

• Paper: https://proceedings.mlr.press/v97/behrmann19a.html
• Code: https://github.com/jhjacobsen/invertible-resnet

(9)Training wide residual networks for deployment using a single bit for each weight

• Paper: https://openreview.net/forum?id=rytNfI1AZ
• Code: https://github.com/McDonnell-Lab/1-bit-per-weight

(10)Residual Attention Network for Image Classification

• Paper: https://arxiv.org/abs/1704.06904
• Code: https://github.com/tengshaofeng/ResidualAttentionNetwork-pytorch

(11)MaxViT: Multi-Axis Vision Transformer

• Paper: https://arxiv.org/abs/2204.01697
• Code: https://github.com/google-research/maxvit

(12)MLP-Mixer: An all-MLP Architecture for Vision

• Paper: https://arxiv.org/abs/2105.01601
• Code: https://github.com/google-research/vision_transformer

(13)A ConvNet for the 2020s

• Paper: https://arxiv.org/abs/2201.03545
• Code: https://github.com/facebookresearch/ConvNeXt

(14)UniNet: Unified Architecture Search with Convolution, Transformer, and MLP

• Paper: https://arxiv.org/abs/2110.04035
• Code: https://github.com/sense-x/uninet

(14)CBAM: Convolutional Block Attention Module

• Paper: https://arxiv.org/abs/1807.06521
• Code: https://github.com/luuuyi/CBAM.PyTorch

(15)Res2Net: A New Multi-scale Backbone Architecture

• Paper: https://arxiv.org/abs/1904.01169
• Code: https://github.com/Res2Net/Res2Net-PretrainedModels

2. object detection

(1)Mask R-CNN

• Paper: https://arxiv.org/abs/1703.06870
• Code:https://github.com/facebookresearch/Detectron

(2)YOLOv3

• Paper: https://arxiv.org/pdf/1804.02767
• Code: https://pjreddie.com/yolo/

(3)MaxViT

• Paper: https://arxiv.org/pdf/2204.01697
• Code: https://github.com/google-research/maxvit

(4)ConvNeXt

• Paper: https://openaccess.thecvf.com/content/CVPR2022/papers/Liu_A_ConvNet_for_the_2020s_CVPR_2022_paper
• Code: https://github.com/facebookresearch/ConvNeXt

(5)ScratchDet

• Paper: https://arxiv.org/pdf/1810.08425.pdf
• Code: https://github.com/KimSoybean/ScratchDet

(6)PoolNet

• Paper: https://openaccess.thecvf.com/content_CVPR_2019/papers/Liu_A_Simple_Pooling-Based_Design_for_Real-Time_Salient_Object_Detection_CVPR_2019_paper.pdf
• Code: https://github.com/backseason/PoolNet

(7)CPD

• Paper: https://openaccess.thecvf.com/content_CVPR_2019/papers/Wu_Cascaded_Partial_Decoder_for_Fast_and_Accurate_Salient_Object_Detection_CVPR_2019_paper.pdf
• Code: https://github.com/wuzhe71/CPD

(8)OPANAS

• Paper: https://openaccess.thecvf.com/content/CVPR2021/papers/Liang_OPANAS_OneShot_Path_Aggregation_Network_Architecture_Search_for_Object_Detection_CVPR_2021_paper.pdf
• Code: https://github.com/VDIGPKU/OPANAS.

(9)RetinaNet

• Paper: https://openaccess.thecvf.com/content_ICCV_2017/papers/Lin_Focal_Loss_for_ICCV_2017_paper.pdf
• Code: https://github.com/yhenon/pytorch-retinanet

(10) U^2Net

• Paper:https://arxiv.org/pdf/2005.09007.pdf
• Code:https://github.com/yhenon/pytorch-retinanet

3. semantic segmentation

(1)FCN

• Paper: https://openaccess.thecvf.com/content_cvpr_2015/papers/Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf
• Code:https://github.com/shekkizh/FCN.tensorflow

(2)U-Net

• Paper: https://arxiv.org/pdf/1505.04597.pdf%EF%BC%89
• Code: https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/

(3)SegNet

• Paper: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7803544
• Code: https://github.com/alexgkendall/caffe-segnet

(4)DiSegNet

• Paper: http://www.mipg.upenn.edu/yubing/DiSegNet.pdf
• Code: https://github.com/Francesco-Voto/Disegnetti

(5)Unet++

• Paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7357299/
• Code: https://github.com/MrGiovanni/UNetPlusPlus

(6)ResUNet-a

• Paper: https://arxiv.org/pdf/1904.00592.pdf
• Code: https://github.com/feevos/resuneta

(7)ConvNet

• Paper: https://ojs.aaai.org/index.php/AAAI/article/view/10510
• Code: https://github.com/TorontoDeepLearning/convnet

(8)Inception-ResNet

• Paper: https://ojs.aaai.org/index.php/aaai/article/view/11231
• Code: https://github.com/titu1994/Inception-v4

(9)DANet

• Paper: https://openaccess.thecvf.com/content_CVPR_2019/papers/Fu_Dual_Attention_Network_for_Scene_Segmentation_CVPR_2019_paper.pdf
• Code: https://github.com/junfu1115/DANet

(10)DeepLab

• Paper: https://arxiv.org/pdf/1606.00915.pdf
• Code: https://github.com/kazuto1011/deeplab-pytorch

(11)DeepLabv3

• Paper: https://arxiv.org/pdf/1706.05587.pdf
• Code: https://github.com/fregu856/deeplabv3

(12)DeepLabv3+

• Paper: https://openaccess.thecvf.com/content_ECCV_2018/papers/Liang-Chieh_Chen_Encoder-Decoder_with_Atrous_ECCV_2018_paper.pdf
• Code: https://github.com/tensorflow/models/tree/master/research/deeplab}{TensorFlow

(13)ESPNet

• Paper: https://scholar.google.com/scholar?hl=zh-CN&as_sdt=0%2C5&q=”Espnet%3A+Efficient+spatial+pyramid+of+dilated+convolutions+for+semantic+segmentation&btnG=&lr=
• Code: https://github.com/sacmehta/ESPNet

(14)ESPNetv2

• Paper: https://openaccess.thecvf.com/content_CVPR_2019/papers/Mehta_ESPNetv2_A_Light-Weight_Power_Efficient_and_General_Purpose_Convolutional_Neural_CVPR_2019_paper.pdf
• Code: https://github.com/sacmehta/ESPNetv2-COREML (15)DCU-Net -none
• Paper: https://link.springer.com/article/10.1007/s11042-022-12418-w
• Code: https://github.com/hwding-whu/DCU-Net

(16)CCNet

• Paper: https://openaccess.thecvf.com/content_ICCV_2019/papers/Huang_CCNet_Criss-Cross_Attention_for_Semantic_Segmentation_ICCV_2019_paper.pdf
• Code: https://github.com/speedinghzl/CCNet

(17)EMANet

• Paper: https://openaccess.thecvf.com/content_ICCV_2019/papers/Li_Expectation-Maximization_Attention_Networks_for_Semantic_Segmentation_ICCV_2019_paper.pdf
• Code: https://github.com/XiaLiPKU/EMANet

(18)Swin Transformer

• Paper: https://openaccess.thecvf.com/content/ICCV2021/papers/Liu_Swin_Transformer_Hierarchical_Vision_Transformer_Using_Shifted_Windows_ICCV_2021_paper.pdf
• Code: https://github.com/microsoft/Swin-Transformer

(19)TransUNet

• Paper: https://arxiv.org/pdf/2102.04306.pdf
• Code: https://github.com/Beckschen/TransUNet

(20)TransGAN

• Paper: https://proceedings.neurips.cc/paper_files/paper/2021/file/7c220a2091c26a7f5e9f1cfb099511e3-Paper.pdf
• Code: https://github.com/VITA-Group/TransGAN

(21)SAM

• Paper: https://scholar.google.com/scholar?hl=zh-CN&as_sdt=0%2C5&q=Seg+anything&btnG=
• Code: https://github.com/facebookresearch/segment-anything

4. Compression

(1)Slimmable Compressive Autoencoders for Practical Neural Image Compression

• Paper: https://arxiv.org/abs/2103.15726
• Code: https://github.com/FireFYF/SlimCAE

(2)MAXIM: Multi-Axis MLP for Image Processing

• Paper: https://arxiv.org/abs/2201.02973
• Code: https://github.com/google-research/maxim

(3)Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules

• Paper: https://arxiv.org/abs/2001.01568
• Code: https://github.com/ZhengxueCheng/Learned-Image-Compression-with-GMM-and-Attention

(4)Online Multi-Granularity Distillation for GAN Compression

• Paper: https://arxiv.org/abs/2108.06908
• Code: https://github.com/mit-han-lab/gan-compression

5. Denoise

(1)Unpaired Learning of Deep Image Denoising

• Paper: https://arxiv.org/abs/2008.13711
• Code: https://github.com/XHWXD/DBSN

(2)D2HNet: Joint Denoising and Deblurring with Hierarchical Network for Robust Night Image Restoration

• Paper: https://arxiv.org/abs/2207.03294
• Code: https://github.com/zhaoyuzhi/D2HNet

(3)Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes

• Paper: https://arxiv.org/abs/2101.06085
• Code: https://github.com/neycyanshi/DDRNet

(4)Recorrupted-to-Recorrupted: Unsupervised Deep Learning for Image Denoising

• Paper: https://openaccess.thecvf.com/content/CVPR2021/html/Pang_Recorrupted-to-Recorrupted_Unsupervised_Deep_Learning_for_Image_Denoising_CVPR_2021_paper.html
• Code: https://github.com/PangTongyao/Recorrupted-to-Recorrupted-Unsupervised-Deep-Learning-for-Image-Denoising

(5)Multi-stage image denoising with the wavelet transform

• Paper: https://arxiv.org/abs/2209.12394
• Code: https://github.com/hellloxiaotian/MWDCNN

(6)Adaptive Consistency Prior based Deep Network for Image Denoising

• Paper:https://openaccess.thecvf.com/content/CVPR2021/papers/Ren_Adaptive_Consistency_Prior_Based_Deep_Network_for_Image_Denoising_CVPR_2021_paper.pdf
• Code: https://github.com/chaoren88/DeamNet

(7)NBNet: Noise Basis Learning for Image Denoising with Subspace Projection

• Paper: https://arxiv.org/abs/2012.15028
• Code: https://github.com/MegEngine/NBNet

6. Deblur

(1)DeFMO: Deblurring and Shape Recovery of Fast Moving Objects

• Paper: https://arxiv.org/abs/2012.00595
• Code: https://github.com/rozumden/DeFMO?tab=readme-ov-file

(2)PSD: Principled Synthetic-to-Real Dehazing Guided by Physical Priors

• Paper: https://openaccess.thecvf.com/content/CVPR2021/papers/Chen_PSD_Principled_Synthetic-to-Real_Dehazing_Guided_by_Physical_Priors_CVPR_2021_paper.pdf
• Code: https://github.com/zychen-ustc/PSD-Principled-Synthetic-to-Real-Dehazing-Guided-by-Physical-Priors

(3)Rethinking Coarse-to-Fine Approach in Single Image Deblurring

• Paper: https://arxiv.org/abs/2108.05054
• Code: https://github.com/chosj95/MIMO-UNet

(4)DarkDeblur: Learning single-shot image deblurring in low-light condition

• Paper: https://www.sciencedirect.com/science/article/abs/pii/S0957417423002403
• Code: https://github.com/sharif-apu/DarkDeblur

(5)Multi-scale frequency separation network for image deblurring

• Paper: https://arxiv.org/abs/2206.00798
• Code: https://github.com/LiQiang0307/MSFS-Net

(6)Self-supervised Non-uniform Kernel Estimation with Flow-based Motion Prior for Blind Image Deblurring

• Paper: https://openaccess.thecvf.com/content/CVPR2023/papers/Fang_Self-Supervised_Non-Uniform_Kernel_Estimation_With_Flow-Based_Motion_Prior_for_Blind_CVPR_2023_paper.pdf
• Code: https://github.com/Fangzhenxuan/UFPDeblur?tab=readme-ov-file

(7)Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes

• Paper: https://arxiv.org/abs/2104.01601
• Code: https://github.com/zzh-tech/RSCD

7. Super-Resolution

(1)Learning Enriched Features for Real Image Restoration and Enhancement

• Paper: https://arxiv.org/abs/2003.06792
• Code: https://github.com/swz30/MIRNet

(2)MemNet: A Persistent Memory Network for Image Restoration

• Paper: https://arxiv.org/abs/1708.02209
• Code: https://github.com/tyshiwo/MemNet

(3)In-Domain GAN Inversion for Real Image Editing

• Paper: https://arxiv.org/pdf/2004.00049
• Code: https://github.com/genforce/idinvert_pytorch

(4)Invertible Image Rescaling

• Paper: https://arxiv.org/abs/2005.05650
• Code: https://github.com/pkuxmq/Invertible-Image-Rescaling

(5)Reconstruction by inpainting for visual anomaly detection

• Paper: https://www.sciencedirect.com/science/article/abs/pii/S0031320320305094
• Code: https://github.com/plutoyuxie/Reconstruction-by-inpainting-for-visual-anomaly-detection

(6)Efficient Long-Range Attention Network for Image Super-resolution

• Paper: https://arxiv.org/abs/2203.06697
• Code: https://github.com/xindongzhang/ELAN

(7)SwinIR: Image Restoration Using Swin Transformer

• Paper: https://arxiv.org/pdf/2108.10257
• Code: https://github.com/JingyunLiang/SwinIR

(8)Uformer: A General U-Shaped Transformer for Image Restoration

• Paper: https://arxiv.org/abs/2106.03106
• Code: https://github.com/ZhendongWang6/Uformer

(9)Residual Feature Distillation Network for Lightweight Image Super-Resolution

• Paper: https://arxiv.org/abs/2009.11551
• Code: https://github.com/njulj/RFDN

Part III: the explanation of skip connection in the residual block

1. Information flow

(1) Residual 3D Scene Flow Learning with Context-Aware Feature Extraction

• Paper: https://arxiv.org/abs/2109.04685
• Code: https://github.com/IRMVLab/CARFlow

(2) Residual 3D Scene Flow Learning with Context-Aware Feature Extraction

• Paper: https://pubmed.ncbi.nlm.nih.gov/35018536/

(3) STRPM: A Spatiotemporal Residual Predictive Model for High-Resolution Video Prediction

• Paper: https://arxiv.org/abs/2203.16084
• Code: https://github.com/ZhengChang467/STRPM

(4) Real Image Denoising with Feature Attention

• Paper: https://arxiv.org/abs/1904.07396
• Code: https://github.com/saeed-anwar/RIDNet

(5) Residual Dense Network for Image Super-Resolution

• Paper: https://arxiv.org/abs/1802.08797
• Code: https://github.com/lingtengqiu/RDN-pytorch

2. Ensemble learning

(1) Progressive Ensemble Kernel-Based Broad Learning System for Noisy Data Classification

• Paper: https://ieeexplore.ieee.org/document/9390392

(2) Residual Networks Behave Like Ensembles of Relatively Shallow Networks

• Paper: https://arxiv.org/abs/1605.06431

3. Regularizations

(1) Visualizing the Loss Landscape of Neural Nets

• Paper: https://arxiv.org/abs/1712.09913
• Code: https://github.com/tomgoldstein/loss-landscape

(2) Skip Connections Eliminate Singularities

• Paper: hhttps://arxiv.org/abs/1701.09175

(3) ShakeDrop Regularization for Deep Residual Learning

• Paper: https://arxiv.org/pdf/1802.02375.pdf

(4) VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks

• Paper: https://arxiv.org/abs/1909.04485

4. Eliminate singularities

(1) Deep Residual Learning for Image Recognition

• Paper: https://arxiv.org/abs/1701.09175

(2) The Shattered Gradients Problem: If resnets are the answer, then what is the question?

• Paper: https://arxiv.org/abs/1702.08591

(3) SRNET: A Shallow Skip Connection Based Convolutional Neural Network Design for Resolving Singularities

• Paper: https://jcst.ict.ac.cn/en/article/doi/10.1007/s11390-019-1950-8
• Code: https://github.com/JiechaoSheng/SRNet

(4) Why Is Everyone Training Very Deep Neural Network With Skip Connections?

• Paper: https://www.semanticscholar.org/paper/Why-Is-Everyone-Training-Very-Deep-Neural-Network-Oyedotun-Ismaeil/93bf04aaba372ce915b4c417068d09d9edfe92ad

5. Keep gradient stable

(1) The Shattered Gradients Problem: If resnets are the answer, then what is the question?

• Paper: https://arxiv.org/abs/1702.08591