Home

Awesome

Active Bias - Training More Accurate Neural Networks by Emphasizing High Variance Samples

Unofficial tensorflow implementation of Active Bias. Specifically, we implemented SGD-WPV (SGD Weighted by Prediction Variance), which is one of the methods using the concept of high variance in the paper. In this repository, we call it Active Bias for convenience.

Publication </br> Chang, H. S., Learned-Miller, E., & McCallum, A., Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples," Advances in Neural Information Processing Systems (NIPS), pp. 1002–1012, 2017.

1. Summary

For more accurate training, Active Bias emphasizes uncertain samples with high prediction variances. As shown in below figure borrowed from the original paper, samples with low variances are too easy or too hard for training, therefore Active Bias focuses on uncertain samples with high variance. That is, the sample having highly variate prediction history is preferred to construct next mini-batch. In detail, Active Bias maintains a history structure to store all softmax outputs (or probabilities) of true labels, and computes the variance of accumulated prababilities for each sample. Then, based on the variance, it reweights the backward loss of the samples in each mini-batch before backward propagation.

<p align="center"> <img src="figures/overview.png " width="650"> </p>

2. Experimental Setup and Network Architecture

3. Environment

4. How to Run

5. Tutorial 1: Comparison of learning curves on clean CIFAR-10 (without any noise)

<p align="center"> <img src="figures/tutorial_1.png " width="650"> </p>

6. Tutorial 2: Comparison of the best test error with varying noise rates.

<p align="center"> <img src="figures/tutorial_2.png " width="400"> </p>