Awesome

Perception Test: A Diagnostic Benchmark for Multimodal Video Models

News

The Second Perception Test Challenge is being organised as an ECCV2024 workshop! Please see the website here for more details and links to eval.ai challenge pages: ptchallenge-workshop.github.io.

Overview

Quickstart visualisation notebook
Dataset Explorer	Dataset Explorer
Download data	Download section here
Evaluation scripts (including data loader, dummy baseline, evaluation metrics)	multiple-choice video QA, object tracking, action localisation, point tracking, sound localisation, grounded video QA
Challenges and evaluation servers	multiple-choice video QA, object tracking, action localisation, point tracking, sound localisation, grounded video QA

Perception Test: A Diagnostic Benchmark for Multimodal Video Models is a multimodal benchmark designed to comprehensively evaluate the perception and reasoning skills of multimodal video models. The Perception Test dataset introduces real-world videos designed to show perceptually interesting situations and defines multiple tasks (object and point tracking, action and sound localisation, multiple-choice and grounded video question-answering) that require understanding of memory, abstract patterns, physics, and semantics, across visual, audio, and text modalities.

In this repository, you will find:

• A summary of the Perception Test and the associated challenge
• A detailed description of the data and annotations in the Perception Test (interactive demo notebook here)
• Details about how to download the data and annotations in the Perception Test (download section here)
• Metrics for evaluating the performance on the different tasks (metrics section here)
• Dummy baselines showcasing how to evaluate models on each of the tasks (baselines section here)

5-minutes summary of the Perception Test

Try the Perception Test for yourself by accessing this quiz.

For more example videos in the Perception Test, check out this playlist.

Download the data and annotations

The Perception Test dataset can be downloaded as zip files containing:

• annotations in JSON format
• videos (including audio) as MP4 files
• audio-only files in WAV format
• pre-computed features for the action localisation and sound localisation tasks.

Full Dataset Splits

*In test videos, where the end of the video gives away the answer to some questions (e.g. in cup-games, where is the hidden object at the end), we cut the end of the video. For the validation split, we provide the frame id where the cut should be made: cut_frame_mapping_valid.json.

Challenge Downloads

Video IDs
Since some of the challenges use subsets of the benchmark, we provide here the lists of video IDs for each challenge. These should be used to filter the videos/audios/annotations from the full splits above. For single object tracking, single point tracking, and grounded video QA we provide separate zip files since the subsets are much smaller than the full dataset.

Single Object Tracking
Challenge link: https://eval.ai/web/challenges/challenge-page/2094/overview

Single Point Tracking
Challenge link: https://eval.ai/web/challenges/challenge-page/2108/overview

Temporal Action Localisation
Challenge link: https://eval.ai/web/challenges/challenge-page/2101/overview

Temporal Sound Localisation
Challenge link: https://eval.ai/web/challenges/challenge-page/2109/overview

Multiple-Choice Video QA
Challenge link: https://eval.ai/web/challenges/challenge-page/2091/overview

Grounded Video QA
Challenge link: https://eval.ai/web/challenges/challenge-page/2110/overview

Baselines

In this repo we provide dummy baselines to demonstrate how to load the data, evaluate and recreate some baseline results from the paper. For the other results in the baselines section in the paper, we will be adding another external repo.

Metrics

Metrics code to evaluate performance for the different tasks coming soon.

Perception Test annotations

Explore the annotations: data_visualisation.ipynb

Summary

Video metadata

Object tracks

Point tracks

Action segments

Sound segments

Multiple-choice video question-answers

Grounded video question-answers

Feedback and support

If you have any questions, feedback, or require support regarding the Perception Test dataset or challenge, please contact us at perception-test@google.com.

Citing this work

@inproceedings{patraucean2023perception,
      title={Perception Test: A Diagnostic Benchmark for Multimodal Video Models}, 
      author={Viorica Pătrăucean and Lucas Smaira and Ankush Gupta and Adrià Recasens Continente and Larisa Markeeva and Dylan Banarse and Skanda Koppula and Joseph Heyward and Mateusz Malinowski and Yi Yang and Carl Doersch and Tatiana Matejovicova and Yury Sulsky and Antoine Miech and Alex Frechette and Hanna Klimczak and Raphael Koster and Junlin Zhang and Stephanie Winkler and Yusuf Aytar and Simon Osindero and Dima Damen and Andrew Zisserman and João Carreira},
      booktitle={Advances in Neural Information Processing Systems},
      year={2023},
      url={https://openreview.net/forum?id=HYEGXFnPoq}
}

License and disclaimer

Copyright 2022 DeepMind Technologies Limited

All software is licensed under the Apache License, Version 2.0 (Apache 2.0); you may not use this file except in compliance with the Apache 2.0 license. You may obtain a copy of the Apache 2.0 license at: https://www.apache.org/licenses/LICENSE-2.0

All other materials are licensed under the Creative Commons Attribution 4.0 International License (CC-BY). You may obtain a copy of the CC-BY license at: https://creativecommons.org/licenses/by/4.0/legalcode

Unless required by applicable law or agreed to in writing, all software and materials distributed here under the Apache 2.0 or CC-BY licenses are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the licenses for the specific language governing permissions and limitations under those licenses.

This is not an official Google product.