Awesome
HandFormer
PyTorch implementation of "On the Utility of 3D Hand Poses for Action Recognition", ECCV 2024.
<img src="https://img.shields.io/badge/arXiv-2403.09805-orange" alt="arXiv Paper"> <img src="https://img.shields.io/badge/Project%20Page-HandFormer-blue" alt="Project Page">
<img src="docs/images/teaser.png" width="90%">Requirements
Data Sources
Assembly101
Poses
Download AssemblyPoses.zip from here. For access, follow the instructions from Assembly101 official repo.
$ unzip AssemblyPoses.zip -d /<path_to_your_data_dir>/assembly101_poses/
RGB
For experiments with RGB, we provide DINOv2 features (at 30fps) for view 4
(Camera ID: C10119_rgb
) of Assembly101 here.
Alternatively, TSM features for any of the views can be downloaded from the same directory.
Frame-wise TSM or DINOv2 features are stored as lmdb files.
Annotations
Download the CSV files from fine-grained annotations and put them into <path_to_your_data_dir>/Assembly101_FG_annotations/
.
Details about the annotations can be found here.
H2O $${\color{green}\text{Coming soon...}}$$
Environment
Create a conda environment with the necessary packages.
$ conda env create -f package_list.yml -n handformer_env
Preprocessing
The parameters and data directories for preprocessing are placed in Preprocess/config.py
. Follow the instructions there to modify and run the following:
$ python Preprocess/1_get_clip_data.py
$ python Preprocess/2_get_final_data.py
Training
All the parameters and data paths for training are defined and explained in train_config.yaml
. The parameters are populated with default values. Modify paths (and parameters, if necessary) and run-
$ python HandFormer/train_model.py
Both Pose-only and Pose+RGB variants can be found in the configuration file. If RGB is used, rgb_feature_source
needs to be specified. Additionally, update data paths in HandFormer/feeders/feeder.py
(within lines 80-115) to refer to the downloaded lmdb files of TSM or DINOv2 features.
Evaluation
To obtain test scores, simply put additional placeholder columns in test.csv
to match train.csv
.
Set parameters in test_config.yaml
and run-
$ python HandFormer/test_model.py
Prepare appropriate output file from the saved scores to submit to the evaluation platforms (e.g., [Assembly101][H2O].)
Acknowledgements
This repository is based on
Thanks for their great work!
Citation
Please cite this work if you find it useful:
@inproceedings{shamil2025utility,
title={On the Utility of 3D Hand Poses for Action Recognition},
author={Shamil, Md Salman and Chatterjee, Dibyadip and Sener, Fadime and Ma, Shugao and Yao, Angela},
booktitle={European Conference on Computer Vision},
pages={436--454},
year={2025},
organization={Springer}
}