Home

Awesome

Learning from Noisy Labels with Deep Neural Networks: A Survey

This is a repository to help all readers who are interested in handling noisy labels.

If your papers are missing or you have other requests, please contact to ghkswns91@gmail.com.</br> We will update this repository and paper on a regular basis to maintain up-to-date.

Citation (.bib) </br>

@article{song2022survey,
  title={Learning from Noisy Labels with Deep Neural Networks: A Survey},
  author={Song, Hwanjun and Kim, Minseok and Park, Dongmin and Shin, Yooju and Jae-Gil Lee},
  journal={IEEE Transactions on Neural Networks and Learning Systems},
  year={2022}}

Contents

<a name="papers"></a>

List of Papers with Categorization

All Papers are sorted chronologically according to five categories below, so that you can find related papers more quickly.

<p align="center"> <img src="files/images/high-level-view.png " width="650"> </p>

We also provide a tabular form of summarization with their methodological comaprison (Table 2 in the paper). - [here] <br/> This is a brief summary for the categorization. Please see Section III in our survey paper for the details - [here]

[Index: Robust Architecture, Robust Regularization, Robust Loss Function, Loss Adjsutment, Sample Selection]

Robust Learning for Noisy Labels
|--- A. Robust Architecture
     |--- A.1. Noise Adaptation Layer: adding a noise adaptation layer at the top of an underlying DNN to learn label transition process
     |--- A.2. Dedicated Architecture: developing a dedicated architecture to reliably support more diverse types of label noises.
|--- B. Robust Regularization
     |--- B.1. Explicit Regularization: an explicit form that modifies the expected tarining loss, e.g., weight decay and dropout.
     |--- B.2. Implicit Regularization: an implicit form that gives the effect of stochasticity, e.g., data augmentation and mini-batch SGD.
|--- C. Robust Loss Function: designing a new loss function robust to label noise.
|--- D. Loss Adjsutment
     |--- D.1. Loss Correction: multiplying the estimated transition matrix to the prediction for all the observable labels.
     |--- D.2. Loss Reweighting: multiplying the estimated example confidence (weight) to the example loss.
     |--- D.3. Label Refurbishment: replacing the original label with other reliable one.
     |--- D.4. Meta Learning: finding an optimal adjustment rule for loss reweighing or label refurbishment.
|--- E. Sample Selection
     |--- E.1. Multi-network Learning: collaborative learning or co-training to identify clean examples from noisy data.
     |--- E.2. Multi-round Learning: refining the selected clean set through training multiple rounds.
     |--- E.3. Hybrid Leanring: combining a specific sample selection strategy with a specific semi-supervised learning model or other orthogonal directions.

In addition, there are some valuable theoretical or empirical papers for understanding the nature of noisy labels.<br> Go to Theoretical or Empirical Understanding.

<a name="A"></a>

A. Robust Architecture

A.1. Noise Adaptation Layer

YearVenueTitleImplementation
2015ICCVWebly supervised learning of convolutional networksOfficial (Caffe)
2015ICLRWTraining convolutional networks with noisy labelsUnofficial (Keras)
2016ICDMLearning deep networks from noisy labels with dropout regularizationOfficial (MATLAB)
2016ICASSPTraining deep neural-networks based on unreliable labelsUnofficial (Chainer)
2017ICLRTraining deep neural-networks using a noise adaptation layerOfficial (Keras)

A.2. Dedicated Architecture

YearVenueTitleImplementation
2015CVPRLearning from massive noisy labeled data for image classificationOfficial (Caffe)
2018NeurIPSMasking: A new perspective of noisy supervisionOfficial (TensorFlow)
2018TIPDeep learning from noisy image labels with quality embeddingN/A
2019ICMLRobust inference via generative classifiers for handling noisy labelsOfficial (PyTorch)

<a name="B"></a>

B. Robust Regularization

B.1. Explicit Regularization

YearVenueTitleImplementation
2018ECCVDeep bilevel learningOfficial (TensorFlow)
2019CVPRLearning from noisy labels by regularized estimation of annotator confusionOfficial (TensorFlow)
2019ICMLUsing pre-training can improve model robustness and uncertaintyOfficial (PyTorch)
2020ICLRCan gradient clipping mitigate label noise?Unofficial (PyTorch)
2020ICLRWasserstein adversarial regularization (WAR) on label noiseN/A
2021ICLRRobust early-learning: Hindering the memorization of noisy labelsOfficial (PyTorch)
2021ICLRWhen Optimizing f-Divergence is Robust with Label NoiseOfficial (PyTorch)
2021ICCVLearning with Noisy Labels via Sparse RegularizationOfficial (PyTorch)
2021NeurIPSOpen-set Label Noise Can Improve Robustness Against Inherent Label NoiseOfficial (PyTorch)

B.2. Implicit Regularization

YearVenueTitleImplementation
2015ICLRExplaining and harnessing adversarial examplesUnofficial (PyTorch)
2017ICLRWRegularizing neural networks by penalizing confident output distributionsUnofficial (PyTorch)
2018ICLRMixup: Beyond empirical risk minimizationOfficial (PyTorch)
2021CVPRAugmentation Strategies for Learning with Noisy LabelsOfficial (PyTorch)
2021CVPRAutoDO: Robust AutoAugment for Biased Data With Label Noise via Scalable Probabilistic Implicit DifferentiationOfficial (PyTorch)

<a name="C"></a>

C. Robust Loss Function

YearVenueTitleImplementation
2017AAAIRobust loss functions under label noise for deep neural networksN/A
2017ICCVSymmetric cross entropy for robust learning with noisy labelsOfficial (Keras)
2018NeurIPSGeneralized cross entropy loss for training deep neural networks with noisy labelsUnofficial (PyTorch)
2020ICLRCurriculum loss: Robust learning and generalization against label corruptionN/A
2020ICMLNormalized loss functions for deep learning with noisy labelsOfficial (PyTorch)
2020ICMLPeer loss functions: Learning from noisy labels without knowing noise ratesOfficial (PyTorch)
2021CVPRLearning Cross-Modal Retrieval with Noisy LabelsOfficial (Pytorch)
2021CVPRA Second-Order Approach to Learning With Instance-Dependent Label NoiseOfficial (PyTorch)
2022ICLRAn Information Fusion Approach to Learning with Instance-Dependent Label NoiseN/A

<a name="D"></a>

D. Loss Adjustment

D.1. Loss Correction

YearVenueTitleImplementation
2017CVPRMaking deep neural networks robust to label noise: A loss correction approachOfficial (Keras)
2018NeurIPSUsing trusted data to train deep networks on labels corrupted by severe noiseOfficial (PyTorch)
2019NeurIPSAre anchor points really indispensable in label-noise learning?Official (PyTorch)
2020NeurIPSDual T: Reducing estimation error for transition matrix in label-noise learningN/A
2021AAAITackling Instance-Dependent Label Noise via a Universal Probabilistic ModelOfficial (PyTorch)

D.2. Loss Reweighting

YearVenueTitleImplementation
2017TNNLSMulticlass learning with partially corrupted labelsUnofficial (PyTorch)
2017NeurIPSActive Bias: Training more accurate neural networks by emphasizing high variance samplesUnofficial (TensorFlow)

D.3. Label Refurbishment

YearVenueTitleImplementation
2015ICLRTraining deep neural networks on noisy labels with bootstrappingUnofficial (Keras)
2018ICMLDimensionality-driven learning with noisy labelsOfficial (Keras)
2019ICMLUnsupervised label noise modeling and loss correctionOfficial (PyTorch)
2020NeurIPSSelf-adaptive training: beyond empirical risk minimizationOfficial (PyTorch)
2020ICMLError-bounded correction of noisy labelsOfficial (PyTorch)
2021AAAIBeyond class-conditional assumption: A primary attempt to combat instancedependent label noiseOfficial (PyTorch)

D.4. Meta Learning

YearVenueTitleImplementation
2017NeurIPSWLearning to learn from weak supervision by full supervisionUnofficial (TensorFlow)
2017ICCVLearning from noisy labels with distillationN/A
2018ICMLLearning to reweight examples for robust deep learningOfficial (TensorFlow)
2019NeurIPSMeta-Weight-Net: Learning an explicit mapping for sample weightingOfficial (PyTorch)
2020CVPRDistilling effective supervision from severe label noiseOfficial (TensorFlow)
2021AAAIMeta label correction for noisy label learningOfficial (PyTorch)
2021ICCVAdaptive Label Noise Cleaning with Meta-Supervision for Deep Face RecognitionN/A

<a name="E"></a>

E. Sample Selection

E.1. Multi-network Learning

YearVenueTitleImplementation
2017NeurIPSDecoupling when to update from how to updateOfficial (TensorFlow)
2018ICMLMentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labelsOfficial (TensorFlow)
2018NeurIPSCo-teaching: Robust training of deep neural networks with extremely noisy labelsOfficial (PyTorch)
2019ICMLHow does disagreement help generalization against label corruption?Official (PyTorch)
2021CVPRJo-SRC: A Contrastive Approach for Combating Noisy LabelsOfficial (PyTorch)

E.2. Single- or Multi-round Learning

YearVenueTitleImplementation
2018CVPRIterative learning with open-set noisy labelsOfficial (Keras)
2019ICMLLearning with bad training data via iterative trimmed loss minimizationOfficial (GluonCV)
2019ICMLUnderstanding and utilizing deep neural networks trained with noisy labelsOfficial (Keras)
2019ICCVO2U-Net: A simple noisy label detection approach for deep neural networksUnofficial (PyTorch)
2020ICMLWHow does early stopping can help generalization against label noise?Official (Tensorflow)
2020NeurIPSA topological filter for learning with label noiseOfficial (PyTorch)
2021ICLRLearning with Instance-Dependent Label Noise: A Sample Sieve ApproachOfficial (PyTorch)
2021NeurIPSFINE Samples for Learning with Noisy LabelsOfficial (PyTorch)
2022ICLRSample Selection with Uncertainty of Losses for Learning with Noisy LabelsN/A

E.3. Hybrid Learning

YearVenueTitleImplementation
2019ICMLSELFIE: Refurbishing unclean samples for robust deep learningOfficial (TensorFlow)
2020ICLRSELF: Learning to filter noisy labels with self-ensemblingN/A
2020ICLRDivideMix: Learning with noisy labels as semi-supervised learningOfficial (PyTorch)
2021ICLRRobust curriculum learning: from clean label detection to noisy label self-correctionN/A
2021NeurIPSUnderstanding and Improving Early Stopping for Learning with Noisy LabelsOfficial (PyTorch)

<a name="F"></a>

Theoretical or Empirical Understanding

How Does a Neural Network’s Architecture Impact Its Robustness to Noisy Labels, NeurIPS 2021 [Link]<br> Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise, AAAI 2021 [Link] <br> Understanding Instance-Level Label Noise: Disparate Impacts and Treatments, ICML 2021 [Link] <br> Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations, ICLR 2022 [Link] <br>

Learning from Noisy Labels towards Realistic Scenarios

There have been some studies to solve more realistic setups associated with noisy labels.