Home

Awesome

XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning

The Cross-lingual Choice of Plausible Alternatives dataset is a benchmark to evaluate the ability of machine learning models to transfer commonsense reasoning across languages. The dataset is the translation and reannotation of the English COPA (Roemmele et al. 2011) and covers 11 languages from 11 families and several areas around the globe. The dataset is challenging as it requires both the command of world knowledge and the ability to generalise to new languages. All the details about the creation of XCOPA and the implementation of the baselines are available in the paper.

Languages | Baselines | Cite | Paper

Data

The XCOPA data are stored in data. Alternatively, they can be loaded through ðŸĪ—Datasets as follows:

from datasets import load_dataset

xcopa_dataset = load_dataset('xcopa')

The "translate test" data obtained via Google Translate are available in the folder data-gmt (note that these do not include Quechua).

Examples

LanguagePremiseQuestionChoice 1Choice 2
quSipasqa cereal mikhunanpi kuruta tarirqan.ResultPayqa pukunman ÃąuqÃąuta churakurqan.Payqa manam mikhuyta munarqanchu.
enThe girl found a bug in her cereal.ResultShe poured milk in the bowl.She lost her appetite.
thāļ•āļēāļ‚āļ­āļ‡āļ‰āļąāļ™āđāļ”āļ‡āđāļĨāļ°āļšāļ§āļĄCauseāļ‰āļąāļ™āļĢāđ‰āļ­āļ‡āđ„āļŦāđ‰āļ‰āļąāļ™āļŦāļąāļ§āđ€āļĢāļēāļ°
enMy eyes became red and puffy.CauseI was sobbing.I was laughing.

Languages

ISO 639-2NameFamilyArea<sup>1</sup>
etEstonianUralicNorthern Europe
htHaitian CreoleFrench CreoleCarribean
idIndonesianAustronesianSoutheastern Asia
itItalianIndo-EuropeanSouthern Europe
quSouthern Quechua<sup>2</sup>QuechuanSouthern America
swSwahiliNiger-CongoEastern Africa
taTamilDravidianSouthern Asia
thThaiKra-DaiSoutheastern Asia
trTurkishTurkicWestern Asia
viVietnameseAustroasiaticSoutheastern Asia
zhMandarin ChineseSino-TibetanEastern Asia

<sup>1</sup> According to the United Nations geoscheme.

<sup>2</sup> Translation by Irma Alvarez Ccoscco, an Eastern Apurímac Quechua speaker.

Leaderboard

If you want to see your results reported, please:

  1. submit them on the XCOPA state-of-the-art page on Papers with Code;
  2. make a pull request changing the table below.
ModelPaperavgethtiditquswtathtrvizh
HumanPonti et al. (2020)97.6098.296.4100.097.094.899.098.698.296.498.496.6
RoBERTa Large (Translate test)Ponti et al. (2020)76.0581.073.882.277.8(50.0)74.279.671.479.681.086.0
XLM-R LargePonti et al. (2020)68.6971.4(50.0)79.872.6(50.0)59.273.072.874.473.878.6
MAD-X BasePfeiffer et al. (2020)60.9461.353.765.863.052.556.361.961.860.366.167.6

The performance of other multilingual pre-trained encoders is shown in the figure.

Cite

If you use the data from this repository, please cite both XCOPA \cite{ponti2020xcopa} and the original COPA paper \cite{roemmele2011choice}.

@inproceedings{ponti2020xcopa,
  title={{XCOPA: A} Multilingual Dataset for Causal Commonsense Reasoning},
  author={Edoardo M. Ponti, Goran Glava\v{s}, Olga Majewska, Qianchu Liu, Ivan Vuli\'{c} and Anna Korhonen},
  booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year={2020},
  url={https://ducdauge.github.io/files/xcopa.pdf}
}

@inproceedings{roemmele2011choice,
  title={Choice of plausible alternatives: An evaluation of commonsense causal reasoning},
  author={Roemmele, Melissa and Bejan, Cosmin Adrian and Gordon, Andrew S},
  booktitle={2011 AAAI Spring Symposium Series},
  year={2011},
  url={https://people.ict.usc.edu/~gordon/publications/AAAI-SPRING11A.PDF},
}