DeepLearnToolbox

A Matlab toolbox for Deep Learning.

Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data. It is inspired by the human brain's apparent deep (layered, hierarchical) architecture. A good overview of the theory of Deep Learning theory is Learning Deep Architectures for AI

For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.

• The Next Generation of Neural Networks (Hinton, 2007)
• Recent Developments in Deep Learning (Hinton, 2010)
• Unsupervised Feature Learning and Deep Learning (Ng, 2011)

If you use this toolbox in your research please cite:

Prediction as a candidate for learning deep hierarchical models of data (Palm, 2012)

Directories included in the toolbox

NN/ - A library for Feedforward Backpropagation Neural Networks

CNN/ - A library for Convolutional Neural Networks

DBN/ - A library for Deep Belief Networks

SAE/ - A library for Stacked Auto-Encoders

CAE/ - A library for Convolutional Auto-Encoders

util/ - Utility functions used by the libraries

data/ - Data used by the examples

tests/ - unit tests to verify toolbox is working

For references on each library check REFS.md

Example: Deep Belief Network

%%  train a 100-100-100 DBN and use its weights to initialize a FFNN
dbn.sizes = [100 100 100];      % Number of neurons in each layer
opts.numepochs =   5;           % Number of full sweeps through data
opts.batchsize = 100;           % Mini-batch size
opts.momentum  =   0;           % Momentum (0 = none)
opts.alpha     =   1;           % Learning rate
dbn = dbnsetup(dbn, train_x, opts);     % Init network
dbn = dbntrain(dbn, train_x, opts);     % Train network

% Use the parameters learned from pre-training to initialize a FFNN to be used for classification
nn.size = [100 100 100];
nn = nnsetup(nn, train_x, train_y);
for i = 1 : 3
    nn.W{i} = dbn.rbm{i}.W;
    nn.b{i} = dbn.rbm{i}.c;
end

nn.alpha  = 1;
nn.lambda = 1e-4;
opts.numepochs =  10;
opts.batchsize = 100;

nn = nntrain(nn, train_x, train_y, opts);
[er, bad] = nntest(nn, test_x, test_y);

printf('%5.2f% error', 100 * er)
figure; visualize(nn.W{1}', 1);         %Visualize the weights in the lowest layer

Example: Stacked Denoising Auto Encoder

%%  ex1 train a 100-100 hidden unit SDAE and use it to initialize a FFNN
%  Setup and train a stacked denoising autoencoder (SDAE)
sae.size = [100 100];
sae = saesetup(sae, train_x);

sae.ae{1}.alpha = 1;      % Learning rate
sae.ae{1}.inl   = 0.5;    % fraction of zero-masked inputs (the noise)
sae.ae{2}.alpha = 1;      % Learning rate
sae.ae{2}.inl   = 0.5;    % fraction of zero-masked inputs (the noise)

opts.numepochs =   5;     % Number of full sweeps through data
opts.batchsize = 100;     % Mini-batch size

sae = saetrain(sae, train_x, opts);     % Do the pre-training

%  use the SDAE to initialize a FFNN
nn.size = [100 100]; 
nn = nnsetup(nn, train_x, train_y);

nn.W{1} = sae.ae{1}.W{1};
nn.b{1} = sae.ae{1}.b{1};
nn.W{2} = sae.ae{2}.W{1};
nn.b{2} = sae.ae{2}.b{1};
nn.lambda = 1e-5;   %  L2 weight decay
nn.alpha  = 1e-0;   %  Learning rate

opts.numepochs =   5;
opts.batchsize = 100;

nn = nntrain(nn, train_x, train_y, opts);

[er, bad] = nntest(nn, test_x, test_y);
printf('%5.2f% error', 100 * er); % Display error rate
figure; visualize(nn.W{1}', 1)   %  Visualize the weights

Setup

1. Download.
2. addpath(genpath('DeepLearnToolbox'));

Everything is work in progress