Home

Awesome

RareAct

This repository contains annotation for the RareAct dataset as well as an evaluation script for computing the wAP and sAP metrics described in the paper.

RareAct

Requirements (only for the evaluation script)

Data

You can download the videos zipped into one file here. The video names are the YouTube ids of the videos.

The annotation file is hosted in the github repo and named as rareact.csv

Here is a description of each column:

Column NameTypeExampleDescription
idint14Unique ID for the annotated video segment.
video_idstring7frRY7aGwMUYouTube ID of the video where the segment originated from (unique per video).
startint3Start time in seconds of the action segment.
endint5End time in seconds of the action segment.
class_idint [0-148]8The class identifier of the actions (verb, noun). Maximum id: 148.
verbstringcutAction verb describing the interaction.
nounstringlaptopObject noun subject of the interaction.
annotationint [0-4]1Annotation for the given clip and (verb, noun) class. 1: Positive. 2: Hard negative (only verb is right): 3: Hard negative (only noun is right). 4: Hard negative (Both verb and noun are valid but verb is not applied to noun). 0: Negative.

Evaluation script

We provide an evaluation python script. To run an evaluation you need first to create a prediction output numpy matrix of shape 7607x149. where each row represent the samples ordered similarly as in rareact.csv and each column is the prediction score for each of the action class_id.

To compute the mWAP just run:

python compute_score.py predictions.npy 

To compute the mSAP (n=100) just run:

python compute_score.py predictions.npy 100 

where predictions.npy is the prediction output numpy array as described above.

References

If you find this dataset useful, please cite the following paper:

@article{miech20rareact,
   title={RareAct: A video dataset of unusual interactions},
   author={Miech, Antoine and Alayrac, Jean-Baptiste and Laptev, Ivan and Sivic, Josef and Zisserman, Andrew},
   journal={arxiv:2008.01018},
   year={2020},
}