Awesome

Task-Oriented Dialogue Research Progress Survey

Content

• Introduction

• Updates

• Call for Contribution

• Leader Boards

• Datasets Introduction

• Acknowledgement

<a name="intro"></a>Introduction

This repo is a dataset and methods survey for Task-oriented Dialogue.

We investigated most existing dialogue datasets and summarized their basic information, such as brief, download link and size.

We also included leader boards of popular dataset to present research progress in the task oriented dialogue fields.

A Chinese intro & news for this project is available here

Refer to this repo:

@misc{MAML_Pytorch,
  author = {Yutai Hou},
  title = {Task-Oriented Dialogue Research Progress Survey},
  year = {2018},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/AtmaHou/Task-Oriented-Dialogue-Research-Progress-Survey/}},
  commit = {master}
}

<a name="updates"></a> Updates

This section records big updates to ease refer (See ./release_detail.md or click links below):

• Updates 2020.7.30: Add 3 new datasets, re-claim 6 abnormal NLU results, refine layout.
• Updates 2020.7.27: Add 10 new datasets, 5 new SOTA results, 1 correctness.

<a name="call"></a> Call for Contributions

Contributions are welcomed, you are encouraged to:

• Directly pull request
• Send me new dataset info
• Send me new experiment results from published paper or public code implements.

<a name="leader"></a> Leader Boards

The ranking is depended on published results of related papers. We are trying to keep it up-to-date. The ranking may be unfair because features used and train/dev set splitting in those papers may be different. However, it shows a trend of research, and would be helpful for someone to start a project about task-oriented dialogue.

Dialogue State Tracking

Dialogue state tacking task aims to predict or give representation of dialogue state, which usually contains a goal constraint, a set of requested slots, and the user's dialogue act.

MultiWOZ 2.0 - Dialogue State Tracking

Multi-Domain Wizard-of-Oz dataset (MultiWOZ), a fully-labeled collection of human-human written conversations spanning over multiple domains and topics. At a size of 10k dialogues, it is at least one order of magnitude larger than all previous annotated task-oriented corpora.

The new, corrected versions of the dataset are available at MultiWOZ 2.1 (2019), MultiWOZ 2.2 (2020).

Notice: Models marked with * are open-vocabulary based models.`

DSTC2 - Dialogue State Tracking

Clarification of dataset types:

The main results we list here are obtained from pure DSTC2 dataset (ASR n-best).

However, we don't list other kinds of DSTC2 data source results such as DSTC2-text (It formulates the dialog state tracking as a machine reading problem which read the dialog transcriptions multiple times and answer the questions about each of the slot, for more info please refer to paper) and DSTC-cleaned (It is used by the NBT paper and fixes ASR noise and typo during training and include ASR noise during testing, The cleaned version is available at here),

NLU: Slot Filling

Slot filling task aims to recognize key entity within user utterance, such as position and time.

Snips - Slot Filling

ATIS - Slot Filling

Notice: The following works have abnormal high-scores, because they are considered to exploit special pre-processing steps: Bi-model-Decoder (Wang et al., 2018), Intent Gating + Self-atten. (Li et al., 2018), Atten.-Based (Liu and Lane, 2016)

NLU: Intent Detection

Intent detection task aims to classify user utterance into different domain or intents.

Snips - Intent Detection

ATIS - Intent Detection

Notice-1: The following works have abnormal high-scores, because they are considered to exploit special pre-processing steps: Bi-model-Decoder (Wang et al., 2018), Intent Gating + Self-atten. (Li et al., 2018), Atten.-Based (Liu and Lane, 2016), BLSTM (Zhang et al., 2016)

<a name="detail"></a> Dataset Introductions

See the data details Here or in Excel File

Following information is included for each dataset:

• Name
• Introduction
• Link (Download & Paper)
• Multi or single turn
• Task detail
• Whether Public Accessible
• Size & Stats
• Included Label
• Missing Label

Tips: The table below may not be displayed completely, scroll right to see more~

<a name="acknowledgment"></a>Acknowledgment

Thanks for supports from my adviser Wanxiang Che.

Thanks for public contributions from: Shuai Lin, JiAnge, Su Zhu, seeledu, Tony Lin, Jason Krone, Libo Qin, HariiHe, Jelle Bosscher, .