Awesome

Action Anticipation using Latent Goal Learning

Code accompanying the IEEE WACV 2022 paper "Action anticipation using latent goal learning"

EPIC-KITCHENS 55

Training

Download RGB features from RULSTM project, specifically this script

https://github.com/fpv-iplab/rulstm/blob/master/RULSTM/scripts/download_data_ek55.sh

and these lines
```
mkdir -p data/ek55/rgb
curl https://iplab.dmi.unict.it/sharing/rulstm/features/rgb/data.mdb -o data/ek55/rgb/data.mdb
```
The data is now in <pwd>/data/ek55/rgb. Next, fetch the training.csv and validation.csv from the RULSTM project ek55 directory

Run the training script - tsnrgb_feat_latent_goal_action_max_current_action.py
Testing on test set
- Fetch the test_seen.csv and test_unseen.csv from the RULSTM project ek55 directory
- The CSV format is different in training.csv and test_seen.csv. For training.csv, the columns are - segment_id, video_id, start_frame, end_frame, verb, noun, action For test_seen/unseen.csv, the columns are - segment_id, video_id, start_frame, end_frame
- We need to train 2 models - one with RGB features as above and another with OBJ features Download OBJ features from RULSTM project, specifically this script
```
mkdir -p data/ek55/obj
curl https://iplab.dmi.unict.it/sharing/rulstm/features/obj/data.mdb -o data/ek55/obj/data.mdb
```
  The data is now in <pwd>/data/ek55/obj.
Run the training script tsnrgb_feat_latent_goal_action_max_current_action.py. Change feat_dim in main() to 352

Breakfast and 50 Salads

I3D features were obtained from this repo for both 50Salads and Breakfast.

Download the data folder, which contains the features and the ground truth labels. (~30GB) (If you cannot download the data from the previous link, try to download it from here)

Then run [i3d_latent_goal_bf.py] (https://github.com/debadityaroy/LatentGoal/blob/main/i3d_latent_goal_bf.py)

We have actions instead of verb and nouns for Breakfast and 50Salads.

Acknowledgment

This research/project is supported in part by the National Research Foundation, Singapore under its AI Singapore Program (AISG Award No: AISG2-RP-2020-016) and the National Research Foundation Singapore under its AI Singapore Program (Award Number: AISG-RP-2019-010).

In case of issues, please write to roy_debaditya [at] ihpc [dot] a-star [dot] edu [dot] sg

Please cite this work if you use this code

@inproceedings{
wacv22,
title={Action anticipation using latent goal learning},
author={Debaditya Roy and Basura Fernando},
booktitle={WACV},
year={2022},
}