Awesome
SLEAP: Social LEAP Estimates Animal Poses
Looking to get some SLEAP?
We've released the next version of LEAP, with full support for multi-animal pose tracking in addition to all of the functionality of the original LEAP.
Check it out now at sleap.ai!
Related software
- DeepLabCut: The popular deep learning framework for pose estimation. Soon to support full multi-animal pose tracking!
- DeepPoseKit: Supports multi-animal pose tracking via top-down centroid detection (e.g., from Ctrax, idTracker, Tractor, etc.).
- Animal Part Tracker: Supports multi-animal pose tracking and lots of other functionality!
LEAP (deprecated)
Full movie: YouTube
This repository contains code for LEAP (LEAP Estimates Animal Pose), a framework for animal body part position estimation via deep learning.
Preprint: Pereira et al., bioRxiv (2018)
We are still working on documentation and preparing parts of the code. See the Features section below for an overview and status of each component.
We recommend starting with the Tutorial: Training Leap From Scratch.
Features
- Tracking and alignment code
- Cluster sampling GUI
- Skeleton creation GUI (
create_skeleton
) - GUI for labeling new dataset (
label_joints
) - Network training through the labeling GUI
- MATLAB (
predict_box.m
) and Python (leap.predict_box
) interfaces for predicting on new data - GUI for predicting on new data
- Training data + labels for main fly dataset used in analyses
- Trained network for predicting on main fly dataset
- Analysis/figure generation code
- Documentation
- Examples of usage
Installation
Pre-requisites
All neural network and GPU functionality is implemented in Python. The library was designed to be easy to use by providing commandline interfaces, but it can also be used programatically if the MATLAB GUIs are not required.
For the Python environment, we recommend Anaconda 5.1.0 with Python 3.6.4. Note that Python 2.x is not supported.
For GPU support, you'll want to first install the CUDA drivers with CuDNN and then install these packages:
pip install -Iv tensorflow-gpu==1.6.0
pip install -Iv keras==2.1.4
See the TensorFlow installation guide for more info.
If you don't have a GPU that supports CUDA, install the regular TensorFlow distribution:
pip install tensorflow
Please note that CPU execution will be MUCH slower (10-20x) than on a GPU. Consider investing in a GPU for your machine if you're thinking of using LEAP routinely.
Automated installation
To get started with using LEAP, open up MATLAB and download the repository:
>> !git clone https://github.com/talmo/leap.git
Then, install the package and add to the MATLAB path:
>> cd leap
>> install_leap
That's it!
To avoid having to run this function every time you start MATLAB, save the MATLAB Search Path after running it (or just add the leap
subfolder to your permanent path).
Manual: MATLAB dependencies
GUIs and analyses are implemented in MATLAB, but is not required for using the neural network functionality implemented in Python.
We use MATLAB R2018a with the following toolboxes: Parallel Computing Toolbox, Statistics and Machine Learning Toolbox, Computer Vision Toolbox, Image Processing Toolbox, Signal Processing Toolbox.
All MATLAB external toolboxes are included in the leap/toolbox
subfolder. Just add the leap
subdirectory to the MATLAB Search Path to access all functionality:
addpath(genpath('leap'))
Manual: Python dependencies
The versions below were used during development of LEAP but other versions will also likely work.
Libraries required are easily installable via the pip package manager:
pip install -Iv numpy==1.14.1
pip install -Iv h5py==2.7.1
pip install -Iv clize==4.0.3
You will also need OpenCV 3 with Python bindings. We recommend using skvark's excellent precompiled packages:
pip install -Iv opencv-python==3.4.0.12
You can install this library as a python package by downloading this git repository:
git clone https://github.com/talmo/leap.git
then typing:
pip install -e leap # installs the leap directory using pip
If you are using anaconda to manage different python environments, it is highly recommended that you use this matlab script to manage your python envs within matlab: condalab (see their readme for how to use it)
Usage
Refer to the Tutorial: Training Leap From Scratch.
Preprocessing
GUI Workflow
- Cluster sampling: Call
cluster_sample
from MATLAB commandline to launch GUI. - Create skeleton: Call
create_skeleton
from MATLAB commandline to launch GUI. - Label data and train: Call
label_joints
from MATLAB commandline to launch GUI. - Batch estimation: Coming soon.
Programmatic API
See leap/training.py
and leap/predict_box.py
for more info.
Contact and more information
Reach out to us via email: Talmo Pereira (talmo@princeton.edu
)