Awesome

ToTTo Dataset

ToTTo is an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description.

During the dataset creation process, tables from English Wikipedia are matched with (noisy) descriptions. Each table cell mentioned in the description is highlighted and the descriptions are iteratively cleaned and corrected to faithfully reflect the content of the highlighted cells.

We hope this dataset can serve as a useful research benchmark for high-precision conditional text generation.

You can find more details, analyses, and baseline results in our paper. You can cite it as follows:

@inproceedings{parikh2020totto,
  title={{ToTTo}: A Controlled Table-To-Text Generation Dataset},
  author={Parikh, Ankur P and Wang, Xuezhi and Gehrmann, Sebastian and Faruqui, Manaal and Dhingra, Bhuwan and Yang, Diyi and Das, Dipanjan},
  booktitle={Proceedings of EMNLP},
  year={2020}
 }

Getting Started

Download the ToTTo data

The ToTTo dataset is released under the Creative Commons Share-Alike 3.0 license.

To download the data from the command line:

 wget https://storage.googleapis.com/totto-public/totto_data.zip
 unzip totto_data.zip

(or alternatively copy the above url into your browser address bar.)

Inside the totto_data directory you should see three files: totto_train_data.jsonl, totto_dev_data.jsonl, and unlabeled_totto_test_data.jsonl for the training, development, and unlabeled test sets respectively.

Download the evaluation scripts

You can find evaluation scripts and some exploratory processing scripts at this repository. It also includes a separate README file with instruction on how to run the evaluation.

Dataset Description

The ToTTo dataset consists of three .jsonl files, where each line is a JSON dictionary with the following format:

{
  "table_page_title": "'Weird Al' Yankovic",
  "table_webpage_url": "https://en.wikipedia.org/wiki/%22Weird_Al%22_Yankovic",
  "table_section_title": "Television",
  "table_section_text": "",
  "table": "[Described below]",
  "highlighted_cells": [[22, 2], [22, 3], [22, 0], [22, 1], [23, 3], [23, 1], [23, 0]],
  "example_id": 12345678912345678912,
  "sentence_annotations": [{"original_sentence": "In 2016, Al appeared in 2 episodes of BoJack Horseman as Mr. Peanutbutter's brother, Captain Peanutbutter, and was hired to voice the lead role in the 2016 Disney XD series Milo Murphy's Law.",
                  "sentence_after_deletion": "In 2016, Al appeared in 2 episodes of BoJack Horseman as Captain Peanutbutter, and was hired to the lead role in the 2016 series Milo Murphy's Law.",
                  "sentence_after_ambiguity": "In 2016, Al appeared in 2 episodes of BoJack Horseman as Captain Peanutbutter, and was hired for the lead role in the 2016 series Milo Murphy's 'Law.",
                  "final_sentence": "In 2016, Al appeared in 2 episodes of BoJack Horseman as Captain Peanutbutter and was hired for the lead role in the 2016 series Milo Murphy's Law."}],
}

The table field is a List[List[Dict]]. The outer lists represents rows and the inner lists columns. Each Dict has the fields column_span: int, is_header: bool, row_span: int, and value: str. The first two rows for the example above look as follows:

[
  [
    {    "column_span": 1,
         "is_header": true,
         "row_span": 1,
         "value": "Year"},
    {    "column_span": 1,
         "is_header": true,
         "row_span": 1,
         "value": "Title"},
    {    "column_span": 1,
         "is_header": true,
         "row_span": 1,
         "value": "Role"},
    {    "column_span": 1,
         "is_header": true,
         "row_span": 1,
         "value": "Notes"}
  ],
  [
    {    "column_span": 1,
         "is_header": false,
         "row_span": 1,
         "value": "1997"},
    {    "column_span": 1,
         "is_header": false,
         "row_span": 1,
         "value": "Eek! The Cat"},
    {    "column_span": 1,
         "is_header": false,
         "row_span": 1,
         "value": "Himself"},
    {    "column_span": 1,
         "is_header": false,
         "row_span": 1,
         "value": "Episode: 'The FugEektive'"}
  ], ...
]

-The table metadata consists of the table_page_title, table_section_title, and table_section_text strings to help give the model more context about the table.

-The highlighted_cells field is a List[[row_index, column_index]] where each [row_index, column_index] pair indicates that table[row_index][column_index] is highlighted.

-The example_id is simply a unique id for this example.

-The sentence_annotations field consists of the original sentence and the sequence of revised sentences performed in order to produce the final_sentence. See our paper for more details.

To help understand the dataset, you can find a sample of the train and dev sets in the sample/ folder of our supplementary repository. It additionally provides the create_table_to_text_html.py script that visualizes examples, the output of which you can also find in the sample/ folder.

Official Task

The official task described in our paper is given the table, highlighted cells, and table metadata (table_page_title, table_section_title, and table_section_text) as input, to generate the final_sentence.

Dev and Test Set

The dev and test set have between two and three references for each example, which are added to the list at the sentence_annotations key. The test set annotations are private and thus not included in the data.

If you want us to evaluate your model on the development or the private test set, please submit your files here. You can find more submission information below. By emailing us or by submitting prediction files, you consent to being contacted by Google about your submission, this dataset or any related competitions.

We provide two splits within the dev and test sets - one uses previously seen combinations of table headers and one uses unseen combinations. The sets are marked using the overlap_subset: bool flag that is added to the JSON representation. By filtering the evaluation to examples with the flag set to true, you will be able to test the generalization ability of your model.

Leaderboard

We are maintaining a leaderboard with official results on our test set.

The leaderboard indicates whether or not a model was trained on any auxiliary Wikipedia data. This is because our tables and (unrevised) test targets are from Wikipedia and thus we would like to study the effect of using additional Wikipedia data to train models.

We ask you to not incorporate any part of the ToTTo development set into the training data, and only use it for validation/hyperparameter tuning as development sets are typically used.

In addition to BLEU and PARENT, we also report a learnt metric BLEURT. The checkpoint used was BLEURT-base-128 which can be found here. To handle multiple references, we take the average of the scores as suggested by Sellam et al. 2020.

Leaderboard Submission

If you want to submit dev and test outputs, please format your predictions as a single .txt file with line-separated predictions. The predictions should be in the same order as the examples in the test.jsonl file. You can upload your prediction files here and email us at totto@google.com to tell us you have submitted. By emailing us or by submitting prediction files, you consent to being contacted by Google about your submission, this dataset or any related competitions.