Awesome

HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors

Paper

Wang, Xiao and Wu, Zongzhen and Jiang, Bo and Bao, Zhimin and Zhu, Lin and Li, Guoqi and Wang, Yaowei and Tian, Yonghong. "HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors." arXiv preprint arXiv:2211.09648 (2022). [arXiv] [Demovideo] [Poster]

Abstract

The main streams of human activity recognition (HAR) algorithms are developed based on RGB cameras which are suffered from illumination, fast motion, privacy-preserving, and large energy consumption. Meanwhile, the biologically inspired event cameras attracted great interest due to their unique features, such as high dynamic range, dense temporal but sparse spatial resolution, low latency, low power, etc. As it is a newly arising sensor, even there is no realistic large-scale dataset for HAR. Considering its great practical value, in this paper, we propose a large-scale benchmark dataset to bridge this gap, termed HARDVS, which contains 300 categories and more than 100K event sequences. We evaluate and report the performance of multiple popular HAR algorithms, which provide extensive baselines for future works to compare. More importantly, we propose a novel spatial-temporal feature learning and fusion framework, termed ESTF, for event stream based human activity recognition. It first projects the event streams into spatial and temporal embeddings using StemNet, then, encodes and fuses the dual-view representations using Transformer networks. Finally, the dual features are concatenated and fed into a classification head for activity prediction. Extensive experiments on multiple datasets fully validated the effectiveness of our model.

News

• :fire: [2023.12.09] Our paper is accepted by AAAI-2024 !!!

• :fire: [2023.05.29] The class label (i.e., category name) is available at [HARDVS_300_class.txt]

• :fire: [2022.12.14] HARDVS dataset is integrated into the SNN toolkit [SpikingJelly]

Demo Videos

• A demo video for the HARDVS dataset can be found by clicking the image below:

• Video Tutorial for this work can be found by clicking the image below:

• Representative samples of HARDVS can be found below:

Dataset Download

• Download from Baidu Disk:

  [Event Images] 链接：https://pan.baidu.com/s/1OhlhOBHY91W2SwE6oWjDwA?pwd=1234    提取码：1234
  [Compact Event file] 链接：https://pan.baidu.com/s/1iw214Aj5ugN-arhuxjmfOw?pwd=1234 提取码：1234
  [RGB Event Images] 链接：https://pan.baidu.com/s/1w-z86PH7mGY0CqVBj_MpNA?pwd=1234 提取码：1234
  [Raw Event file] To be updated 

• Download from DropBox:

  To be updated ... 

Environment

conda create -n event python=3.8 pytorch=1.10 cudatoolkit=11.3 torchvision -c pytorch -y
conda activate event
pip3 install openmim
mim install mmcv-full
mim install mmdet  # optional
mim install mmpose  # optional
pip3 install -e .

Details of each package:

<img src="https://github.com/Event-AHU/HARDVS/blob/main/figures/package.jpg" width="400px" >

Our Proposed Approach

An overview of our proposed ESTF framework for event-based human action recognition. It transforms the event streams into spatial and temporal tokens and learns the dual features using multi-head self-attention layers. Further, a FusionFormer is proposed to realize message passing between the spatial and temporal features. The aggregated features are added with dual features as the input for subsequent TF and SF blocks, respectively. The outputs will be concatenated and fed into MLP layers for action prediction.

Train & Test & Evaluation

# train
  CUDA_VISIBLE_DEVICES=0 python tools/train.py configs/recognition/hardvs_ESTF/hardvs_ESTF.py --work-dir path_to_checkpoint --validate --seed 0 --deterministic --gpu-ids=0

# test
  CUDA_VISIBLE_DEVICES=0 python tools/test.py configs/recognition/hardvs_ESTF/hardvs_ESTF.py  path_to_checkpoint --eval top_k_accuracy

Citation

If you find this work useful for your research, please cite the following paper and give us a :star2:.

@article{wang2022hardvs,
  title={HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors},
  author={Wang, Xiao and Wu, Zongzhen and Jiang, Bo and Bao, Zhimin and Zhu, Lin and Li, Guoqi and Wang, Yaowei and Tian, Yonghong},
  journal={arXiv preprint arXiv:2211.09648},
  url={https://arxiv.org/abs/2211.09648}, 
  year={2022}
}

Acknowledgement and Other Useful Materials

• MMAction2: https://github.com/open-mmlab/mmaction2
• SpikingJelly: https://github.com/fangwei123456/spikingjelly