Awesome
Action Recognition with Deep Learning
This branch hosts the code for the technical report "Towards Good Practices for Very Deep Two-stream ConvNets", and more.
Updates
- Jul 16, 2016
- New working sample: "Real-time Action Recognition with Enhanced Motion Vector CNNs" on CVPR 2016.
- Apr 27, 2016
- cuDNN v5 support, featuring the super fast WINOGrad Convolution and cuDNN implementation of BatchNormalization.
- Dec 23, 2015
- Refactored cudnn wrapper to control overall memory consumption. Will automatically find the best algorithm combination under memory constraint.
- Dec 17, 2015
- cuDNN v4 support: faster convolution and batch normalization (around 20% performance gain).
- Nov 22, 2015
- Now python layer can expose a
prefetch()
method, which will be run in parallel with network processing.
- Now python layer can expose a
Features
VideoDataLayer
for inputing video data.- Training on optical flow data.
- Data augmentation with fixed corner cropping and multi-scale cropping.
- Parallel training with multiple GPUs.
- Newest cuDNN integration.
- Slim memory footprints in both training and testing,
Usage
See more in Wiki.
Generally it's the same as the original caffe. Please see the original README. Please see following instruction for accessing features above. More detailed documentation is on the way.
- Video/optic flow data
- First use the optical flow extraction tool to convert videos to RGB images and opitcal flow images.
- A new data layer called
VideoDataLayer
has been added to support multi-frame input. See the UCF101 sample for how to use it. - Note: The
VideoDataLayer
can only input the optical-flow images generated by the tool listed above.
- Fixed corner cropping augmentation
- Set
fix_crop
totrue
intranform_param
of network's protocol buffer definition.
- Set
- "Multi-scale" cropping augmentation
- Set
multi_scale
totrue
intransform_param
- In
transform_param
, specifyscale_ratios
as a list of floats smaller than one, default is[1, .875, .75, .65]
- In
transform_param
, specifymax_distort
to an integer, which will limit the aspect ratio distortion, default to1
- Set
- cuDNN v5
- The cuDNN v5 wrapper has optimized engines for convolution and batch normalization.
- The solver protobuf config has a parameter
richness
which specifies the total GPU memory in MBs available to the cudnn convolution engine as workspaces. Defaultrichness
is 300 (300MB). Using this parameter you can control the GPU memory consumption of training, the system will find the best setup under the memory limit for you. - Training with multiple GPUs
- Requires OpenMPI > 1.7.4 (Why?). Remember to compile your OpenMPI with option
--with-cuda
- Specify list of GPU IDs to be used for training, in the solver protocol buffer definition, like
device_id: [0,1,2,3]
- Compile using cmake and use
mpirun
to launch caffe executable, like
- Requires OpenMPI > 1.7.4 (Why?). Remember to compile your OpenMPI with option
mkdir build && cd build
cmake .. -DUSE_MPI=ON
make && make install
mpirun -np 4 ./install/bin/caffe train --solver=<Your Solver File> [--weights=<Pretrained caffemodel>]
Note: actual batch_size will be num_device
times batch_size
specified in network's prototxt.
- Runtime memory optimization
- Memory optimization drastically reduces memory usage (half for training and almost all for testing) by safely sharing underlying storage of a series of blobs. Note in this case, the Python/Matlab interfaces can no longer retrieve correct contents of affected blobs.
- Training time memory optimization is automatically enabled.
- To adjust memory optimization setting, add 'optimize_mem' option to the network prototxt. It can be set to
TRAIN_ONLY
(default),ALL_OPTIM
, andNO_OPTIM
. - Testing time optimization is disabled by default. To enable testing time optimization, set
optimize_mem
toNO_OPTIM
- To disable memory optimization, set
optimize_mem
toNO_OPTIM
. This may be useful when working with intermediate blobs.
Working Examples
- Actionness Estimation Using Hybrid FCNs
- Real-time Action Recognition with Enhanced Motion Vector CNNs
- Action recognition on UCF101
- Scene recognition on Places205
Extension
Currently all existing data layers sub-classed from BasePrefetchingDataLayer
support parallel training. If you have newly added layer which is also sub-classed from BasePrefetchingDataLayer
, simply implement the virtual method
inline virtual void advance_cursor();
Its function should be forwarding the "data cursor" in your data layer for one step. Then your new layer will be able to provide support for parallel training.
Questions
Contact
Citation
You are encouraged to also cite the following report if you find this repo helpful
@article{MultiGPUCaffe2015,
author = {Limin Wang and
Yuanjun Xiong and
Zhe Wang and
Yu Qiao},
title = {Towards Good Practices for Very Deep Two-Stream ConvNets},
journal = {CoRR},
volume = {abs/1507.02159},
year = {2015},
url = {http://arxiv.org/abs/1507.02159},
}
Following is the original README of Caffe.
Caffe
Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by the Berkeley Vision and Learning Center (BVLC) and community contributors.
Check out the project site for all the details like
- DIY Deep Learning for Vision with Caffe
- Tutorial Documentation
- BVLC reference models and the community model zoo
- Installation instructions
and step-by-step examples.
Please join the caffe-users group or gitter chat to ask questions and talk about methods and models. Framework development discussions and thorough bug reports are collected on Issues.
Happy brewing!
License and Citation
Caffe is released under the BSD 2-Clause license. The BVLC reference models are released for unrestricted use.
Please cite Caffe in your publications if it helps your research:
@article{jia2014caffe,
Author = {Jia, Yangqing and Shelhamer, Evan and Donahue, Jeff and Karayev, Sergey and Long, Jonathan and Girshick, Ross and Guadarrama, Sergio and Darrell, Trevor},
Journal = {arXiv preprint arXiv:1408.5093},
Title = {Caffe: Convolutional Architecture for Fast Feature Embedding},
Year = {2014}
}