Awesome

RecurrentWhisperer - A general class template for training recurrent neural networks using Tensorflow.

RecurrentWhisperer is a base class for training recurrent neural networks or other deep learning models using TensorFlow. RecurrentWhisperer provides functionality for:

1. Training a recurrent neural network using modern techniques for encouraging stable training, such as adaptive learning rate management and adaptive gradient norm clipping. RecurrentWhisperer handles common tasks like splitting training data into batches, making gradient steps based on individual batches of training data, periodically evaluating validation data, and periodically saving model checkpoints.

2. Managing Tensorboard visualizations of training progress.

3. Managing a directory structure for maintaining many different variants of a model (i.e., with different hyperparameter settings). Previously saved models can be readily restored from checkpoints, and training runs can be readily resumed if their execution was interrupted or preempted.

If you are using RecurrentWhisperer in research to be published, please cite our accompanying paper in your publication:

Golub and Sussillo (2018), "FixedPointFinder: A Tensorflow toolbox for identifying and characterizing fixed points in recurrent neural networks," Journal of Open Source Software, 3(31), 1003, https://doi.org/10.21105/joss.01003 .

Recommended Installation

The recommended installation is to assemble all dependencies in a virtual environment.

To create a new virtual environment, enter at the command line:

$ python3 -m venv --system-site-packages your-virtual-env-name

where your-virtual-env-name is a path to the the virtual environment you would like to create (e.g.: /home/rwhisp). Then activate your new virtual environment:

$ source your-virtual-env-name/bin/activate

Next, install all dependencies in your virtual environment.

$ pip install -e git+https://github.com/mattgolub/recurrent-whisperer.git@master#egg=v1.5.0

When you are finished working in your virtual environment, enter:

$ deactivate

Advanced Users

Advanced Python users may skip the Recommended Installation, opting to instead clone this repository and ensure that compatible versions of the following prerequisites are available:

• TensorFlow (recommended version: 2.8; requires at least version 1.14; versions beyond 2.8 are not currently supported) (install).
• NumPy, SciPy and Matplotlib (install SciPy stack, contains both)
• PyYaml (install)

Getting started

See FlipFlop.py for an example subclass that inherits from RecurrentWhisperer for the purposes of training an RNN to implement an N-bit memory.