Awesome

The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction

<a href="https://alexandrosstergiou.github.io/project_pages/TemPr/index.html">[Project page 🌐]</a> <a href="http://arxiv.org/abs/2204.13340">[ArXiv preprint 📃]</a> <a href="https://youtu.be/dcmd8U47BT8">[Video 🎞️]</a>

This is the code implementation for the CVPR'23 paper <a href="http://arxiv.org/abs/2204.13340">The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction</a>.

Abstract

Early action prediction deals with inferring the ongoing action from partially-observed videos, typically at the outset of the video. We propose a bottleneck-based attention model that captures the evolution of the action, through progressive sampling over fine-to-coarse scales. Our proposed <b>Tem</b>poral <b>Pr</b>ogressive (TemPr) model is composed of multiple attention towers, one for each scale. The predicted action label is based on the collective agreement considering confidences of these towers. Extensive experiments over four video datasets showcase state-of-the-art performance on the task of Early Action Prediction across a range of encoder architectures. We demonstrate the effectiveness and consistency of TemPr through detailed ablations.

Dependencies

Ensure that the following packages are installed in your machine:

• adaPool (version >= 0.2)
• coloredlogs (version >= 14.0)
• dataset2database (version >= 1.1)
• einops (version >= 0.4.0)
• ffmpeg-python (version >=0.2.0)
• imgaug (version >= 0.4.0)
• opencv-python (version >= 4.2.0.32)
• ptflops (version >= 0.6.8)
• torch (version >= 1.9.0)
• torchinfo (version >= 1.5.4)
• youtube-dl (version >= 2020.3.24)

You can install the available PyPi packages with the command below:

$ pip install coloredlogs dataset2database einops ffmpeg-python imgaug opencv-python ptflops torch torchvision youtube-dl

and compile the adaPool package as:

$ git clone https://github.com/alexandrosstergiou/adaPool.git && cd adaPool-master/pytorch && make install
--- (optional) ---
$ make test

Datasets

A custom format is used for the train/val label files of each datasets:

This can be done through the scripts provided in labels

We have tested our code over the following datasets:

• UCF-101 : [link]
• Somethong-Something (sub21/v2) : [link]
• EPIC-KITCHENS-100 : [link]
• NTU-RGB : [link]

Videos and image-based datasets

Based on the format that the dataset is stored on disk two options are supported by the repo:

• Videos being stored in video files (e.g. .mp4,.avi,etc.)
• Videos being stored in folders containing their frames in image files (e.g. .jpg)

By default it is assumed that the data are in video format however, you can overwrite this by setting the use_frames call argument to True/true.

Data directory format

We assume a fixed directory formatting that should be of the following structure:

<data>
|
└───<dataset>
        |
        └─── <class_i>
        │     │
        │     │─── <video_id_j>
        │     │         (for datasets w/ videos saved as frames)
        │     │         │
        │     │         │─── frame1.jpg
        │     │         └─── framen.jpg
        │     │    
        │     │─── <video_id_j+1>
        │     │         (for datasets w/ videos saved as frames)
        │     │         │
        │     │         │─── frame1.jpg
        │     │         └─── framen.jpg
       ...   ...

Usage

Training for each of the datasets is done through the homonym .yaml configuration scripts in configs.

You can also use the argument parsers in train.py and inference.py for custom arguments.

Examples

Train on UCF-101 with observation ratio 0.3, 3 scales, with movinet backbone, with the pretrained UCF-101 backbone checkpoint stored in weights, and over 4 gpus:

python train.py --video_per 0.3 --num_samplers 3 --gpus 0 1 2 3 --precision mixed --dataset UCF-101 --frame_size 224 --batch_size 64 --data_dir data/UCF-101/ --label_dir /labels/UCF-101 --workers 16 --backbone movinet --end_epoch 70 --pretrained_dir weights/UCF-101/movinet_ada_best.pth

Run inference over something-something v2 with TemPr and adaptive ensemble over a single gpu with checkpoint file my_chckpt.pth:

python inference.py --config config/inference/smthng-smthng/config.yml --head TemPr_h --pool ada --gpus 0 --pretrained_dir my_chckpt.pth

Calling arguments (for both train.py & inference.py)

The following arguments are used and can be included at the parser of any training script.

Checkpoints

UCF-101

SSsub21

Citation

@inproceedings{stergiou2023wisdom,
    title = {The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction},
    author = {Stergiou, Alexandros and Damen, Dima},
    booktitle = {IEEE/CVF Computer Vision and Pattern Recognition (CVPR)},
    year = {2023}
}

License

MIT