Home

Awesome

Situated Interactive MultiModal Conversations (SIMMC) Challenge 2020

Welcome to the Situated Interactive Multimodal Conversations (SIMMC) Track for DSTC9 2020.

The SIMMC challenge aims to lay the foundations for the real-world assistant agents that can handle multimodal inputs, and perform multimodal actions. We thus focus on task-oriented dialogs that encompass a situated multimodal user context in the form of a co-observed image or virtual reality (VR) environment. The context is dynamically updated on each turn based on the user input and the assistant action. Our challenge focuses on our SIMMC datasets, both of which are shopping domains: (a) furniture (grounded in a shared virtual environment) and, (b) fashion (grounded in an evolving set of images).

Organizers: Ahmad Beirami, Eunjoon Cho, Paul A. Crook, Ankita De, Alborz Geramifard, Satwik Kottur, Seungwhan Moon, Shivani Poddar, Rajen Subba

<figure> <img src="./figures/teaser.png" width="300" alt="Example from SIMMC" align="center"> <figcaption><i>Example from SIMMC-Furniture Dataset</i></figcaption> </figure>

Latest News

Note: DSTC9 SIMMC Challenge was conducted on SIMMC v1.0. Thus all the results and baseline performances are on SIMMC v1.0.

Important Links

Timeline

DateMilestone
June 22, 2020Training & development data released
Sept 28, 2020Test-Std data released, End of Challenge Phase 1
Oct 5, 2020Entry submission deadline, End of Challenge Phase 2
Oct 12, 2020Final results announced

Track Description

Tasks and Metrics

We present three sub-tasks primarily aimed at replicating human-assistant actions in order to enable rich and interactive shopping scenarios.

Sub-Task #1Multimodal Action Prediction
GoalTo predict the correct Assistant API action(s) (classification)
InputCurrent user utterance, Dialog context, Multimodal context
OutputStructural API (action & arguments)
MetricsAction Accuracy, Attribute Accuracy, Action Perplexity
Sub-Task #2Multimodal Dialog Response Generation & Retrieval
GoalTo generate Assistant responses or retrieve from a candidate pool
InputCurrent user utterance, Dialog context, Multimodal context, (Ground-truth API Calls)
OutputAssistant response utterance
MetricsGeneration: BLEU-4, Retrieval: MRR, R@1, R@5, R@10, Mean Rank
Sub-Task #3Multimodal Dialog State Tracking (MM-DST)
GoalTo track user belief states across multiple turns
InputCurrent user utterance, Dialogue context, Multimodal context
OutputBelief state for current user utterance
MetricsSlot F1, Intent F1

Please check the task input file for a full description of inputs for each subtask.

Evaluation

For the DSTC9 SIMMC Track, we will do a two phase evaluation as follows.

Challenge Period 1: Participants will evaluate the model performance on the provided devtest set. At the end of Challenge Period 1 (Sept 28), we ask participants to submit their model prediction results and a link to their code repository.

Challenge Period 2: A test-std set will be released on Sept 28 for the participants who submitted the results for the Challenge Period 1. We ask participants to submit their model predictions on the test-std set by Oct 5. We will announce the final results and the winners on Oct 12.

Challenge Instructions

(1) Challenge Registration

(2) Download Datasets and Code

$ git lfs install
$ git clone https://github.com/facebookresearch/simmc.git

(3) Reporting Results for Challenge Phase 1

(4) Reporting Results for Challenge Phase 2

Contact

Questions related to SIMMC Track, Data, and Baselines

Please contact simmc@fb.com, or leave comments in the Github repository.

DSTC Mailing List

If you want to get the latest updates about DSTC9, join the DSTC mailing list.

Citations

If you want to publish experimental results with our datasets or use the baseline models, please cite the following articles:

@article{moon2020situated,
  title={Situated and Interactive Multimodal Conversations},
  author={Moon, Seungwhan and Kottur, Satwik and Crook, Paul A and De, Ankita and Poddar, Shivani and Levin, Theodore and Whitney, David and Difranco, Daniel and Beirami, Ahmad and Cho, Eunjoon and Subba, Rajen and Geramifard, Alborz},
  journal={arXiv preprint arXiv:2006.01460},
  year={2020}
}

@article{crook2019simmc,
  title={SIMMC: Situated Interactive Multi-Modal Conversational Data Collection And Evaluation Platform},
  author={Crook, Paul A and Poddar, Shivani and De, Ankita and Shafi, Semir and Whitney, David and Geramifard, Alborz and Subba, Rajen},
  journal={arXiv preprint arXiv:1911.02690},
  year={2019}
}

NOTE: The paper above describes in detail the datasets, the NLU/NLG/Coref annotations, and some of the baselines we provide in this challenge. The paper reports the results from an earlier version of the dataset and with different train-dev-test splits, hence the baseline performances on the challenge resources will be slightly different.

License

SIMMC is released under CC-BY-NC-SA-4.0, see LICENSE for details.