Convolutional Neural Network.

By virture of being here, it is assumed that you have gone through the Quick Start.

Building a convolutional neural network is just as similar as an MLNN. The convolutional-pooling layer or convpool layer could be added using the following statement:

net.add_layer ( type = "conv_pool",
                origin = "input",
                id = "conv_pool_1",
                num_neurons = 40,
                filter_size = (5,5),
                pool_size = (2,2),
                activation = ('maxout', 'maxout', 2),
                batch_norm = True,
                regularize = True,
                verbose = verbose
            )

Here the layer has 40 filters each 5X5 followed by batch normalization followed by a maxpooling of 2X2 all with stride 1. The activation used is maxout with maxout by 2. A simpler relu layer could be added thus,

net.add_layer ( type = "conv_pool",
            origin = "input",
            id = "conv_pool_1",
            num_neurons = 40,
            filter_size = (5,5),
            pool_size = (2,2),
            activation = 'relu',
            verbose = verbose
            )

Refer to the APIs for more details on the convpool layer. It is often useful to visualize the filters learnt in a CNN, so we introduce the visualizer module here along with the CNN tutorial. The visualizer can be setup using the add_module method of net object.

net.add_module ( type = 'visualizer',
             params = visualizer_params,
             verbose = verbose
            )

where the visualizer_params is a dictionary of the following format.

visualizer_params = {
            "root"       : 'lenet5',
            "frequency"  : 1,
            "sample_size": 144,
            "rgb_filters": True,
            "debug_functions" : False,
            "debug_layers": False,
            "id"         : 'main'
                }

root is the location where the visualizations are saved, frequency is the number of epochs for which visualizations are saved down, sample_size number of images are saved each time. rgb_filters make the filters save in color. Along with the activities of each layer for the exact same images as the data itself, the filters of neural network are also saved down. For more options of parameters on visualizer refer to the visualizer documentation .

The full code for this tutorial with additional commentary can be found in the file pantry.tutorials.lenet.py. This tutorial runs a CNN for the lenet dataset. If you have toolbox cloned or downloaded or just the tutorials downloaded, Run the code using,

Notes

This code contains three methods.
  1. A modern reincarnation of LeNet5 for MNIST.
  2. The same Lenet with batchnorms
    2.a. Batchnorm before activations. 2.b. Batchnorm after activations.

All these methods are setup for MNIST dataset.

Todo

Add detailed comments.

pantry.tutorials.lenet.lenet5(dataset=None, verbose=1)[source]

This function is a demo example of lenet5 from the infamous paper by Yann LeCun. This is an example code. You should study this code rather than merely run it.

Warning

This is not the exact implementation but a modern re-incarnation.

Parameters:
  • dataset – Supply a dataset.
  • verbose – Similar to the rest of the dataset.
pantry.tutorials.lenet.lenet_maxout_batchnorm_after_activation(dataset=None, verbose=1)[source]

This is a version with nesterov momentum and rmsprop instead of the typical sgd. This also has maxout activations for convolutional layers, dropouts on the last convolutional layer and the other dropout layers and this also applies batch norm to all the layers. The difference though is that we use the batch_norm layer to apply batch norm that applies batch norm after the activation fo the previous layer. So we just spice things up and add a bit of steroids to lenet5(). This also introduces a visualizer module usage.

Parameters:
  • dataset – Supply a dataset.
  • verbose – Similar to the rest of the dataset.
pantry.tutorials.lenet.lenet_maxout_batchnorm_before_activation(dataset=None, verbose=1)[source]

This is a version with nesterov momentum and rmsprop instead of the typical sgd. This also has maxout activations for convolutional layers, dropouts on the last convolutional layer and the other dropout layers and this also applies batch norm to all the layers. The batch norm is applied by using the batch_norm = True parameters in all layers. This batch norm is applied before activation as is used in the original version of the paper. So we just spice things up and add a bit of steroids to lenet5(). This also introduces a visualizer module usage.

Parameters:
  • dataset – Supply a dataset.
  • verbose – Similar to the rest of the dataset.