Home

Awesome

event_utils

Event based vision utility library. For additional detail, see the thesis document Motion Estimation by Focus Optimisation: Optic Flow and Motion Segmentation with Event Cameras. If you use this code in an academic context, please cite:

@PhDThesis{Stoffregen20Thesis,
  author        = {Timo Stoffregen},
  title         = {Motion Estimation by Focus Optimisation: Optic Flow and Motion Segmentation with Event Cameras},
  school        = {Department of Electrical and Computer Systems Engineering, Monash University},
  year          = 2020
}

This is an event based vision utility library with functionality for focus optimisation, deep learning, event-stream noise augmentation, data format conversion and efficient generation of various event representations (event images, voxel grids etc).

The library is implemented in Python. Nevertheless, the library is efficient and fast, since almost all of the hard work is done using vectorisation or numpy/pytorch functions. All functionality is implemented in numpy and pytorch, so that on-GPU processing for hardware accelerated performance is very easy.

The library is divided into eight sub-libraries:

└── lib
    ├── augmentation
    ├── contrast_max
    ├── data_formats
    ├── data_loaders
    ├── representations
    ├── transforms
    ├── util
    └── visualization

augmentation

While the data_loaders learning library contains some code for tensor augmentation (such as adding Gaussian noise, rotations, flips, random crops etc), the augmentation library allows for these operations to occur on the raw events. This functionality is contained within event_augmentation.py.

event_augmentation.py

The following augmentations are available:

Augmentation examples

Some examples of augmentations on the slider_depth sequence from the event camera dataset can be seen above (events in red and blue with the first events in black to show scene structure). (a) the original event stream, (b) doubling the events by adding random correlated events, (c) doubling the events by adding fully random (normal distribution) events, (d) halving the events by removing random, (e) flipping the events horizontally, (f) rotating the events 45 degrees. Demo code to reproduce these plots can be found by executing the following (note that the events need to be in HDF5 format): python lib/augmentation/event_augmentation.py /path/to/slider_depth.h5 --output_path /tmp

contrast_max

The focus optimisation library contains code that allows the user to perform focus optimisation on events. The important files of this library are: events_cmax.py This file contains code to perform focus optimisation. The most important functionality is provided by:

Focus Optimisation

Examples can be seen in the images above: each set of events is drawn with the variance objective function (w.r.t. optic flow motion model) underneath. This set of tools allows optimising the objective function to recover the motion parameters (images generated with the library).

objectives.py

This file implements various objective functions described in this thesis as well as some other commonly cited works. Objective functions inherit from the parent class objective_function. The idea is to make it as easy as possible to add new, custom objective functions by providing a common API for the optimisation code. This class has several members that require initialisation:

warps.py

This file implements warping functions described in this thesis as well as some other commonly cited works. Objective functions inherit from the parent class warp_function. The idea is to make it as easy as possible to add new, custom warping functions by providing a common API for the optimisation code. Initialisation requires setting member variables:

data_formats

The data_formats provides code for converting events in one file format to another. Even though many candidates have appeared over the years (rosbag, AEDAT, .txt, hdf5, pickle, cuneiform clay tablets, just to name a few), a universal storage option for event based data has not yet crystallised. Some of these data formats are particularly useful within particular operating systems or programming languages. For example, rosbags are the natural choice for C++ programming with the ros environment. Since they also store data in an efficient binary format, they have become a very common storage option. However, they are notoriously slow and impractical to process in Python, which has become the de-facto deep-learning language and is commonly used in research due to the rapid development cycle. More practical (and importantly, fast) options are the hdf5 and numpy memmap formats. hdf5 is a more compact and easily accessible format, since it allows for easy grouping and metadata allocation, however it's difficulty in setting up multi-threading access and subsequent buggy behaviour (even in read-only applications) means that memmap is more common for deep learning, where multi-threaded data-loaders can significantly speed up training.

event_packagers.py

The data_formats library provides a packager abstract base class, which defines what a packager needs to do. packagerobjects receive data (events, frames etc) and write them to the desired file format (eg hdf5). Converting file formats is now much easier, since input files now need only to be parsed and the data sent to the packagerwith the appropriate function calls. The functions that need to implemented are:

h5_to_memmap.py and rosbag_to_h5.py

The library implements two converters, one for hdf5 to memmap and one for rosbag to hdf5. These can be easily called from the command line with various options that can be found in the documentation.

add_hdf5_attribute.py

add_hdf5_attribute.py allows the user to add or modify attributes to existing hdf5 files. Attributes are the manner in which metadata is saved in hdf5 files.

read_events.py

read_events.py contains functions for reading events from hdf5 and memmap. The functions are:

data_loader

The deep learning code can be found in the data_loaderslibrary. It contains code for loading events and transforming them into voxel grids in an efficient manner as well as code for data augmentation. Actual networks and cost functions described in this thesis are not implemented in the library but at the project page for that paper.

data_loaders provides a highly versatile pytorch dataloader, which can be used across various storage formats for events (.txt, hdf5, memmap etc). As a result it is very easy to implement new dataloader for a different storage format. The output of the dataloader was originally to provide voxel grids of the events, but can be used just as well to output batched events, due to a custom pytorchcollation function. As a result, the dataloader is useful for any situation in which it is desirable to iterate over the events in a storage medium and is not only useful for deep learning. For instance, if one wants to iterate over the events that lie between all the frames of a davis sequence, the following code is sufficient:

dloader = DynamicH5Dataset(path_to_events_file)
for item in dloader:
	print(item[`events'].shape)

base_dataset.py

This file defines the base dataset class (BaseVoxelDataset), which defines all batching, augmentation, collation and housekeeping code. Inheriting classes (one per data format) need only to implement the abstract functions for providing events, frames and other data from storage. These abstract functions are:

representations

This library contains code for generating representations from the events in a highly efficient, gpu ready manner. Representations Various representations can be seen above with (a) the raw events, (b) the voxel grid, (c) the event image, (d) the timestamp image.

voxel_grid.py

This file contains several means for forming and viewing voxel grids from events. There are two versions of each function, representing a pure numpy and a pytorch implementation. The pytorch implementation is necessary for gpu processing, however it is not as commonly used as numpy, which is so frequently used as to barely be a dependency any more. Functions for pytorch are:

image.py

image.py contains code for forming images from events in an efficient manner. The functions allow for forming images with both discrete and floating point events using bilinear interpolation. Images currently supported are event images and timestamp images using either numpy or pytorch. Functions are:

util

This library contains some utility functions used in the rest of the library. Functions include:

visualisation

The visualization library contains methods for generating figures and movies from events. The majority of figures shown in the thesis were generated using this library. Two rendering backends are available, the commonly used matplotlib plotting library and mayavi, which is a VTK based graphics library. The API for both of these is essentially the same, the main difference being the dependency on matplotlib or mayavi. matplotlib is very easy to set up, but quite slow, mayavi is very fast but more difficult to set up and debug. I will describe the matplotlib version here, although all functionality exists in the mayavi version too (see the code documentation for details).

draw_event_stream.py

The core work is done in this file, which contains code for visualising events and voxel grids for examples). The function for plotting events is plot_events. \input{figures/appendix/visualisations/fig.tex} Input parameters for this function are: