Awesome
AutoNN
AutoNN is a functional wrapper for MatConvNet, implementing automatic differentiation.
It builds on MatConvNet's low-level functions and Matlab's math operators, to create a modern deep learning API with automatic differentiation at its core. The guiding principles are:
- Concise syntax for fast research prototyping, mixing math and deep network blocks freely.
- No boilerplate code to create custom layers, implemented as Matlab functions operating on GPU arrays.
- Minimal execution kernel for backpropagation, with a focus on speed.
Compared to the SimpleNN and DagNN wrappers for MatConvNet, AutoNN is less verbose and has lower computational overhead.
Requirements
- A recent Matlab (preferably 2016b onwards, though older versions may also work).
- MatConvNet (preferably the most recent version, though others may still work).
AutoNN in a nutshell
Defining an objective function with AutoNN is as simple as:
% define inputs and learnable parameters
x = Input() ;
y = Input() ;
w = Param('value', randn(1, 100)) ;
b = Param('value', 0) ;
% combine them using math operators, which define the prediction
prediction = w * x + b ;
% define a loss
loss = sum(sum((prediction - y).^2)) ;
% compile and run the network
net = Net(loss) ;
net.eval({x, rand(100, 1), y, 0.5}) ;
% display parameter derivatives
net.getDer(w)
AutoNN also allows you to use MatConvNet layers and custom functions.
Here's a simplified 20-layer ResNet:
images = Input() ;
% initial convolution
x = vl_nnconv(images, 'size', [3 3 3 64], 'stride', 4) ;
% iterate blocks
for k = 1:20
% compose a residual block, based on the previous output
res = vl_nnconv(x, 'size', [3 3 64 64], 'pad', 1) ; % convolution
res = vl_nnbnorm(res) ; % batch-normalization
res = vl_nnrelu(res) ; % ReLU
% add it to the previous output
x = x + res ;
end
% pool features across spatial dimensions, and do final prediction
pooled = mean(mean(x, 1), 2) ;
prediction = vl_nnconv(pooled, 'size', [1 1 64 1000]) ;
All of MatConvNet's layer functions are overloaded, as well as a growing list of Matlab math operators and functions. The derivatives for these functions are defined whenever possible, so that they can be composed to create differentiable models. A full list can be found here.
Finally, there are several classes to aid training, such as standard datasets, solvers, models, and statistics plotting. It is easy to mix them, and you retain full control over the training loop. For example:
% load dataset
dataset = datasets.CIFAR10('/data/cifar') ;
% create solver
solver = solvers.Adam() ;
solver.learningRate = 0.0001 ;
for epoch = 1:100 % iterate epochs
for batch = dataset.train() % iterate batches
% draw samples
[images, labels] = dataset.get(batch) ;
% evaluate network to compute gradients
net.eval({'images', images, 'labels', labels}) ;
% take one gradient descent step
solver.step(net) ;
end
end
Documentation
Tutorial
The easiest way to learn more is to follow this short tutorial. It covers all the basic concepts and a good portion of the API.
Help pages
Comprehensive documentation is available by typing help autonn
into the Matlab console. This lists all the classes and methods, with short descriptions, and provides links to other help pages.
Converting SimpleNN/DagNN models
For a quicker start or to load pre-trained models, you may want to import them from the existing wrappers. Check help Layer.fromDagNN
.
Examples
The examples
directory has heavily-commented samples. These can be grouped in two categories:
-
The minimal examples (in
examples/minimal
) are very short and self-contained. They are scripts so you can inspect and explore the resulting variables in the command window. -
The full examples (in
examples/cnn
andexamples/rnn
) demonstrate usage of the AutoNN training packages. These include several standard solvers (e.g. Adam, AdaGrad), CNN models (including automatically downloading pre-trained models), and datasets (e.g. ImageNet, CIFAR-10). You can override the parameters on the command window, and experiment with different models and settings.
Screenshots
Some gratuitous screenshots, though the important bits are in the code above really:
Training diagnostics plot
Graph topology plot
Authors
AutoNN was developed by João F. Henriques at the Visual Geometry Group (VGG), University of Oxford.
We gratefully acknowledge contributions by: Sam Albanie, Ryan Webster, Ankush Gupta, David Novotny, Aravindh Mahendran, Stefano Woerner.