Home

Awesome

Action Modifiers

Code and data for the CVPR 2020 paper 'Action Modifiers: Learning from Adverbs in Instructional Videos'.

Data

Training/Test Splits

The files containing the adverb annotations can be found in train.csv and test.csv. The files contain the following columns:

Column NameTypeExampleDescription
idint955Unique id for this adverb-action annotation
vid_idstringS7wF6S5ywo4YouTube id for the video the annotation is for
weak_timestampfloat19.435Value in seconds of the action-adverb in the narration
clustered_adverbstringquicklyAnnotated adverb
clustered_actionstringcutAnnotated action
task_numint105259The id for the task in the HowTo100M dataset
adverbstringfastThe original adverb from the narration
actionstringsliceThe original action from the narration

Features

The features can be downloaded here: https://drive.google.com/open?id=12POBotvtWimAv-PtRswCbUWYucUJ8Aic

This contains two files per entry in train.csv or test.csv, one for RGB features, one for flow features.

Files are named <annotation_id>_<modality>.npz.

Videos

The videos can be downloaded using: python utils/download_videos.py <train.csv|test.csv> <download_dir> --trim 20

The --trim 20 argument extracts 20 seconds around the weak timestamp as used to extract features.

Other useful files

antonym.csv lists each adverb and its antonym

adverb_clusters.csv lists the clusters of adverbs with the following columns:

Column NameTypeExampleDescription
adverb_idint0ID of this adverb
cluster_keystringcoarselyMain adverb representing the cluster
adverbslist of strings['coarsley', 'coarse', 'thickly', 'not finely', 'not fine']Narrated adverbs in this cluster

action_clusters.csv is defined similarly

Code

Training

To train the model run:

python train.py --feature-dir <path_to_directory_containing_features> --checkpoint-dir <path_to_save_checkpoints_to>

To train the model without first training the action embedding run

python train.py --no-pretrain-action --temporal-agg <sdp|average|single> --feature_dir <path_to_directory_containing_features> --checkpoint-dir <path_to_save_checkpoints_to>

Testing

To test a model run:

python test.py --laod <checkpoint_path> --temporal-agg <sdp|average|single> --feature-dir <path_to_features>

Models

Models corresponding to results in the paper can be found under models/ they are:

Subtitle Parsing

To parse subtitles for action-adverb pairs you first need to download the subtitles and punctuated texts. Alternatively you can punctuate your own subtitles with this tool

Then run:

python get_action_adverb_pairs.py <path_to_subtitles> <path_to_punctuated texts> output.csv --adverb-file data/adverbs.csv --action-file data/actions.csv --task-list data/tasks.csv

--adverb-file, --action-file and --task-list are optional arguments use to filter the search space.