Awesome

RealTime QA: What's the Answer Right Now?

Introduction

RealTime QA is a dynamic question answering (QA) platform that inquires about the present. We announce ~30 questions and evaluate real-time baseline systems (e.g., GPT-3 and T5) on a regular basis (weekly in this version).

<img src="https://github.com/realtimeqa/realtimeqa_public/blob/main/figs/twitter.png" height="30" alt="twitter"> This Week's Questions

Data and Retrieval Results

This repository provides our past data, as well as the latest, ongoing questions.

The backnumber questions preceded our real-time baselines that started from June 17, 2022. See our paper for more detail. Our baseline retrieval results (Google custom search and DPR) are also provided (e.g., June 17, 2022 Google custom search). The past questions have all six baseline results under the multiple-choice and original settings (e.g., GPT-3 + Google Custom Search Generation Results).

Submit

The submission window closes when the next set of questions is announced (3 am GMT on every Saturday). Submit via this Google form. The indexes for multiple-choice answer should start with 0. See examples.

Our 6 Real-time Baselines

See our script that we use for the six baseline models in the paper. We run them every week. The results are updated every week on our website.

Citations

@inproceedings{
kasai2023realtime,
title={RealTime {QA}: What's the Answer Right Now?},
author={Jungo Kasai and Keisuke Sakaguchi and yoichi takahashi and Ronan Le Bras and Akari Asai and Xinyan Velocity Yu and Dragomir Radev and Noah A. Smith and Yejin Choi and Kentaro Inui},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2023},
url={https://openreview.net/forum?id=HfKOIPCvsv}
}