Home

Awesome

Second-Moment Loss

The second-moment loss (SML) is a novel training objective for dropout-based regression networks that yields improved uncertainty estimates. The code accompanies our paper "Second-Moment Loss: A Novel Regression Objective for Improved Uncertainties" and allows to reproduce the results therein.

Overview

Baselines

Model training with the second-moment loss is compared to various approaches for uncertainty estimation:

Installation

Conda Virtual Environment

We used a conda environment on Linux Debian Version 9. Use sml.yml to create this environment as follows:

conda env create --name sml --file=sml.yml

Datasets

In order to rerun our evaluation, you need to add the following UCI datasets to the data folder that is located in the same directory as the Jupyter notebooks.

The toy datasets "toy_hf" and "toy_modulated" are generated by calling load_dataset (for argument id='toy_hf' and id='toy_modulated', respectively) in the model_training_and_evaluation.ipynb notebook.

Quick Start

Model Training

The code for the experiments on the UCI and toy datasets is provided in the Jupyter notebook model_training_and_evaluation.ipynb. Make sure to have the conda environment activated. Then execute all cells except the last one.

In the last cell insert a path in line 24 (exp_dir= ...). The path points to a directory exp_dir in which experiment log files are stored. If the directory does not exist it will be created.

To run the training/evaluation for the hyperparameter study (beta parameter) replace the line

methods = available_methods  

with

methods = { 'mc_mod_sml1', 'mc_mod_sml25', 'mc_mod_sml', 'mc_mod_sml75', 'mc_mod_sml90'}

This allows to start the training with the SML loss using different values for beta.

Run the last cell to start the experiment. After the training finishes, exp_dir contains several subdirectories for each dataset. Those subdirectories contain zipped dictionaries that contain the performance/uncertainty metrics on train/test data, see next section.

Model Evaluation

To visualize evaluation results, another notebook is provided: model_evaluation_and_plots.ipynb.

Due to random train-test splits and random initializations, the obtained results might slightly deviate from those reported in the paper.

Used Hardware & Runtimes

All experiments are conducted on a Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz. For object detection we additionally used a Tesla V100 GPU.

Conducting the described experiments with cross validation takes 6h (on CPU) for toy data, 80h (on CPU) for UCI regression and 8h (on GPU) for object regression.

License

The SML code is released under the MIT license.

Citing the Second-Moment Loss

If you use or reference the second-moment loss in your research, please use the following BibTeX entry.

@article{sml,
author = {Joachim Sicking and Maram Akila and Maximilian Pintz and Tim Wirtz and Asja Fischer},
title = {Second-Moment Loss: A Novel Regression Objective for Improved Uncertainties},
year = {2020}
}