Home

Awesome

LSTM Pose Machines

<img src="figs/img.png" width="800" height="300" />

This repo includes the source code of the paper: "LSTM Pose Machines" (CVPR'18) by Yue Luo, Jimmy Ren, Zhouxia Wang, Wenxiu Sun, Jinshan Pan, Jianbo Liu, Jiahao Pang, Liang Lin.

Contact: Yue Luo (lawy623@gmail.com)

Before Everything

You can click the images below to watch our results on video-based pose estimation. The first one is the comparison with the state-of-the-art single image pose estimation method "Convolutional Pose Machines(CPMs)" on videos. Second one is our LSTM Pose Machines on video pose estimation.

<a href="https://youtu.be/gOUcV4ZCT4A" target="_blank"><img src="figs/compare.png" width="400" height="250" /></a> <a href="https://youtu.be/-sP3LWl6Ul0" target="_blank"><img src="figs/result.png" width="400" height="250" /></a>

Prerequisites

The code is tested on 64 bit Linux (Ubuntu 14.04 LTS). You should also install Matlab (R2015a) and OpenCV (At least 2.4.8). We have tested our code on GTX TitanX with CUDA8.0+cuDNNv5. Please install all these prerequisites before running our code.

Installation

  1. Get the code.

    git clone https://github.com/lawy623/LSTM_Pose_Machines.git
    cd LSTM_Pose_Machines
    
  2. Build the code. Please follow Caffe instruction to install all necessary packages and build it.

    cd caffe/
    # Modify Makefile.config according to your Caffe installation/. Remember to allow CUDA and CUDNN.
    make -j8
    make matcaffe
    
  3. Prepare data. We write all data and labels into .mat files.

Training

   ## To run the training matlab scripts from terminal
   sh prototxt/PENN/LSTM_5/train_LSTM.sh   #To trained on PENN dataset
   ## Or
   sh prototxt/sub-JHMDB/LSTM_5_Sub1/train_LSTM.sh   #To trained on sub-JHMDB subset 1, change `line 10` of `video_train_JHMDB.m` to be `modelID = 1` first.
   sh prototxt/sub-JHMDB/LSTM_5_Sub2/train_LSTM.sh   #To trained on sub-JHMDB subset 2, change `line 10` of `video_train_JHMDB.m` to be `modelID = 2` first.
   sh prototxt/sub-JHMDB/LSTM_5_Sub3/train_LSTM.sh   #To trained on sub-JHMDB subset 3, change `line 10` of `video_train_JHMDB.m` to be `modelID = 3` first.

Testing

  ## PENN Dataset
  Head    R_Shoulder L_Shoulder R_Elbow   L_Elbow   R_Wrist   L_Wrist   R_Hip    L_Hip    R_Knee    L_Knee   R_Ankle   L_Ankle || Overall
  98.90%  98.50%     98.60%     96.60%    96.60%    96.60%    96.50%    98.20%   98.20%   97.90%    98.50%   97.30%    97.70%  || 97.73%
  
  ## sub-JHMDB Dataset
  Neck     Belly    Head     R_Shoulder L_Shoulder  R_Hip    L_Hip    R_Elbow  L_Elbow  R_Knee   L_Knee  R_Wrist  L_Wrist  R_Ankle  L_Ankle || Overall
  99.20%   98.97%   98.27%   96.67%     96.13%      98.83%   98.63%   90.17%   89.10%   96.40%   94.80%  85.93%   86.17%   91.90%   89.90%  || 94.09%

To get the results in our paper, you need to remove unlisted joints, calculate average and reorder the accuracy.

Visualization

Citation

Please cite our paper if you find it useful for your work:

@inproceedings{Luo2018LSTMPose,
    title={LSTM Pose Machines},
    author={Yue Luo, Jimmy Ren, Zhouxia Wang, Wenxiu Sun, Jinshan Pan, Jianbo Liu, Jiahao Pang, Liang Lin},
    booktitle={CVPR},
    year={2018},
}