Awesome
This repository is a direct adaptation from the Ego4D git page. Please consider referring their datatset.
The changes are made in ego4d/lta_models.py
New additions -
Aggregrator:
TransformerAggregator_clip (for CLIP_img)
TransformerAggregator_clip_text (for CLIP attention)
Decoder:
MultiHeadDecoder_clip
Encoder_Decoder:
ForecastingEncoderDecoder_clip
ForecastingEncoderDecoder_clip_text
Installation
This code requires Python>=3.7 (this a requirement of pytorch video). If you are using Anaconda, you can create a clean virtual environment with the required Python version with the following command:
conda create -n ego4d_forecasting python=3.7
To proceed with the installation, you should then activate the virtual environment with the following command:
conda activate ego4d_forecasting
Run the following commands to install the requirements:
cat requirements.txt | xargs -n 1 -L 1 pip install
In order to make the ego4d
module loadable, you should add the current directory to the Python path:
export PYTHONPATH=$PWD:$PYTHONPATH
Please note that the command above is not persistent and hence you should run it every time you open a new shell.
Long-Term Action Anticipation
This README reports information on how to train and test the baseline model for the Long-Term Action Anticipation task part of the forecasting benchmark of the Ego4D dataset. The following sections discuss how to download and prepare the data, download the pre-trained models and train and test the different components of the baseline.
Data and models
Download all necessary data and model checkpoints using the Ego4D CLI tool and link the necessary files to the project directory.
# set download dir for Ego4D
export EGO4D_DIR=/path/to/Ego4D/
# download annotation jsons, clips and models for the FHO tasks
python -m ego4d.cli.cli \
--output_directory=${EGO4D_DIR} \
--datasets annotations clips lta_models \
--benchmarks FHO
# link data to the current project directory
mkdir -p data/long_term_anticipation/annotations/ data/long_term_anticipation/clips_hq/
ln -s ${EGO4D_DIR}/v1/annotations/* data/long_term_anticipation/annotations/
ln -s ${EGO4D_DIR}/v1/clips/* data/long_term_anticipation/clips_hq/
# link model files to current project directory
mkdir -p pretrained_models
ln -s ${EGO4D_DIR}/v1/lta_models/* pretrained_models/
The data/long_term_anticipation/annotations
directory should contain the following files
fho_lta_train.json
fho_lta_val.json
fho_lta_test_unannotated.json
fho_lta_taxonomy.json
Where fho_lta_train.json
, fho_lta_val.json
and fho_lta_test_unannotated.json
contain the training, validation and test annotations, respectively, and fho_lta_taxonomy.json.json
contains the verb/noun class id to text mapping.
Downsampling video clips
To allow dataloaders to load clips efficiently, we will downsample video clips to 320p using ffmpeg. The script can be found at tools/long_term_anticipation/resize_clips.sh
and can be run in parallel as a SLURM array job. Remember to adjust the paths and SLURM parameters before running.
sbatch tools/long_term_anticipation/resize_clips.sh
This will create and populate data/long_term_anticipation/clips/
with downsampled clips
Training
We provide code and instructions to train the baseline model. Any of these steps can be skipped and the pretrained model can be used instead (e.g., to avoid generating a pre-trained recognition model from scratch).
See each script for options and details including how to select a specific backbone (SlowFast vs. MViT), how to select different model components (backbones, aggregator modules, and heads) and whether to run locally (to debug) or on the cluster. Running the scripts unaltered will produce the SlowFast-Transformer baseline model. Other options are left in comments.
Train an Ego4D recognition backbone model
bash tools/long_term_anticipation/ego4d_recognition.sh checkpoints/recognition/
The pretrained checkpoints for the recognition models can be found at:
pretrained_models/long_term_anticipation/ego4d_slowfast8x8.ckpt
pretrained_models/long_term_anticipation/ego4d_mvit16x4.ckpt
Train an Ego4D long term anticipation model
bash tools/long_term_anticipation/ego4d_forecasting.sh checkpoints/forecasting/
See script for different model configurations (backbones, aggregator modules, and heads). Tensorboard logs plot training and validation metrics over time. The pretrained checkpoints for the long-term anticipation models can be found at:
pretrained_models/long_term_anticipation/lta_slowfast_concat.ckpt
pretrained_models/long_term_anticipation/lta_slowfast_trf.ckpt
pretrained_models/long_term_anticipation/lta_mvit_concat.ckpt
Generate predictions
Model predictions on the test set can be generated using the following script. See script for option and details.
# Generate model predictions (outputs.json)
bash tools/long_term_anticipation/evaluate_forecasting.sh output/
Citing CLIP+Baseline
@misc{das2022CLIP_EGO,
doi = {10.48550/ARXIV.2207.00579},
url = {https://arxiv.org/abs/2207.00579},
author = {Das, Srijan and Ryoo, Michael S.},
title = {Video + CLIP Baseline for Ego4D Long-term Action Anticipation},
publisher = {arXiv},
year = {2022}
}