Home

Awesome

Remark: The code is updated from the ICML version. The ICML version corresponds to a commit on May 25, 2018.

ResFGB

This is a Theano(>=1.0.0) implementation of "Functional gradient boosting based on residual network perception".

ResFGB is a functional gradient boosting method for learning a resnet-like deep neural network for non-linear classification problems. The model is composed of a linear classifier such as logistic regression and support vector machine, and a feature extraction. In each iteration, these components are trained by alternate optimization, that is, a linear classifier is trained to classify obtained samples through a feature extraction and this extraction map is updated by stacking a resnet-type layer to move samples along the direction of increasing the linear separability. We finally obtain a highly non-linear classifier forming a residual network.

Usage

A simple pseudocode is provided below.

Note: (X,Y): training data, (Xv,Yv): validation data, (Xt,Yt): test data. These are numpy arrays. n_data: the number of training data, input_dim: dimension of the input space, n_class: the number of classes. A label set should be an integer sequence starting with zero.

from resfgb.models import ResFGB, get_hyperparams

hparams = get_hyperparams( n_data, input_dim, n_class )
model = ResFGB( **hparams )
best_iters,_ ,_ = model.fit( X, Y, Xv, Yv, use_best_iter=True )

train_loss, train_acc = model.evaluate( X, Y )
print( 'train_loss: {0}, train_acc: {1}'.format(train_loss, train_acc) )

test_loss, test_acc  = model.evaluate( Xt,  Yt )
print( 'test_loss : {0}, test_acc : {1}'.format(test_loss, test_acc) )

See examples/sample_resfgb.py for more detail.

Hyperparameters

Hyperparameters of ResFGB are mainly divided three types: the first is for learning a linear classifier, the second is for learning a multi-layer network as a resblock, and the other is for the functional gradient method.

The hyperparameters are listed below. 'Default' is a value set by the function resfgb.models.get_hyperparams. input_dim and n_class stand for the dimension of the input space and the number of classes, respectively.

For the linear model

For the resblock

For the functional gradient method