Home

Awesome

SNN for Space

Table of contents

About this project

This project investigates the feasibility of deploying spiking neural networks (SNN) in land cover and land use classification tasks. In particular, a VGG-16 -based artificial neural network (ANN) classifier is converted into an SNN using KerasSpiking. After fine-tuning of the converted model, we compare the accuracy depreciation as well as a potential improvement in the energy consumption on selected hardware platforms.

Requirements

This project uses Python 3.10, and requires the following third-party libraries:

No issues related to using different versions of the other libraries was encountered. Nonetheless, recommend creating a separate environment for this project and installing the versions of the packages specified above. NumPy can be acquired by installing the Anaconda distribution of Python.

Alternatively, all the required libraries can be installed using pip:

pip install numpy==1.19.5 tensorflow==2.6.0 tensorflow-io==0.21.0 tensorflow-datasets==4.5.0 keras-spiking==0.3.0

In case there are issues with running the TensorFlow files on a GPU-equipped workstation, we recommend trying TensorFlow nightly:

pip install tf-nightly tensorflow-nightly-io

Installation

To install this project either download it as a .zip file and extract it into a desired directory or clone it via the terminal or console command:

git clone https://github.com/AndrzejKucik/SNN4Space.git
git clone git@github.com:AndrzejKucik/SNN4Space.git

Usage

Change the current working directory to be the SNN4Spacedirectory.

Training a VGG16-based model on EuroSAT RGB and UC Merced datasets

In the first part of the project we download either the EuroSAT: Land Use and Land Cover Classification with Sentinel-2 Dataset (10 classes, 27000 examples) or the UC Merced Land Use Dataset (21 classes, 100 examples each) . We slice it into the training, validation, and test sets using ratios 80%-10%-10% (theUC Merced data examples are ordered according to version 2.0.0 in TensorFlow Datasets ). We augment the training set using random dihedral group transformation, random crop, random brightness change, random contrast change, random hue change, random saturation change. We use a modified version of the VGG-16 network trained on the ImageNet dataset (parameters from the Keras-TensorFlow) version to construct a classifier for this dataset. We replace the max pooling layers with average pooling layers, we remove the head of the network (all the layers following the last pooling layers) and replace it with a global pooling layer, and a dense classifier without bias. We resize the UC Merced images to (224, 224, 3) shape (to be compatible with the usual VGG-16 input size). The model is trained using the RMSprop optimizer, using early stopping and reducing the learning rate on a plateau (by a factor of 10) if there is no significant improvement in the validation loss after 100 and 50 consecutive epochs respectively. Optionally, L<sub>2</sub> and L<sub>1</sub> regularization is applied to convolutional kernels and biases, respectively.

To train the network run:

python train_models.py [-ds dataset] [-s seed] [-e epochs] [-bs batch_size] [-drpt dropout] [-kl2 kernel_regularizer]
 [-bl1 bias_regularizer] [-lz lower_zoom] [-uz upper_zoom] [-mbd max_brightness_delta] [-mhd max_hue_delta] [-lc 
 lower_contrast] [-uc upper_contrast] [-ls lower_saturation] [-us upper_saturation]

where the optional arguments are:

The default values of these parameters (apart from the batch size) are the ones that empirically gave us the best test accuracy performance (91.43%) on UC Merced and (95. 07%) on EuroSat.

Note that the obtained test set accuracy might be different each time (sometimes even failing to converge), even if identical parameters are used, because the global random seed does not seem to affect the shuffling processes or data augmentation parameters.

Training a spiking model

A VGG-16 -based classifier trained on the EuroSat or UC Merced datasets can be converted into a spiking neural network and trained using KerasSpiking module by running

python train_spiking_models.py -md model_path -ds dataset [-s seed] \
[-i iterate] [-e epochs] [-bs batch_size] [-lr learning_rate] [-t timesteps] \
[-dt dt] [-l2 l2] [-lhz lower_hz] [-uhz upper_hz] [-tau tau] [-lz lower_zoom] \
[-uz upper_zoom] [-mbd max_brightness_delta] [-mhd max_hue_delta] \
[-lc lower_contrast] [-uc upper_contrast] [-ls lower_saturation] \ 
[-us upper_saturation]

where

The rest of the options are as defined the in the previous subsection above, however, epochs, batch_size, and learning_rate are always the starting values if the training is done iteratively, and timesteps is the approximate number of target simulation timesteps (the actual number of timesteps is always a power of 2), while the starting number of timesteps is 1.

Before the training, the local average pooling layers are removed, and the preceding convolutions have their strides set to 2, and their weights appropriately adjusted for consistency. The ReLU activation functions are swapped with spiking activations followed by a low-pass filter

ANN to SNN conversion

The trained spiking model can be fine-tuned for a specific number of time steps and fixed temporal dimension by running:

python fine_tune_snn.py -wp weights_path -ds dataset [-s seed] [-e epochs] \
[-bs batch_size] [-lr learning_rate] [-t timesteps] [-dt dt] [-l2 l2] \
[-lhz lower_hz] [-uhz upper_hz] [-tau tau] [-lz lower_zoom] [-uz upper_zoom] \
[-mbd max_brightness_delta] [-mhd max_hue_delta] [-lc lower_contrast] \
[-uc upper_contrast] [-ls lower_saturation] [-us upper_saturation]

where weights_path is a path to a valid .h5 file with pre-trained weights (as output by train_spiking_models. py), and the rest of the parameters are as above.

Sample spikes

Energy consumption estimation

To compare the estimated energy consumption of the ANN and SNN (both with local pooling layers replaced by doubly-strided convolutions) on representative hardware (standard and neuromorphic), run:

python energy_estimation.py -wp weights_path -ds dataset -bs batch_size \
-t timesteps -dt dt -v verbose

where verbose is the verbosity mode, and the remaining options are as in the [previous section] (#training-a-spiking-model).

Accuracy and energy

License

Property of the European Space Agency. Distributed under MIT licence. See LICENSE for more information.

Contributors

Contact

E-mail: andrzej.kucik@esa.int gabriele.meoni@esa.int

Project Link: https://github.com/AndrzejKucik/SNN4Space

Acknowledgements