Awesome
<p align="center"> <br> <img src="https://user-images.githubusercontent.com/31640436/130242523-edc285f2-beda-4b10-ba2b-6ca5b5d9b6a8.png" width="27%"> <br> </p> <h4 align="center"> <p> <a href="#download">Download</a> | <a href="#baselines-and-evaluation">Baselines and Evaluation</a> | <a href="#results">Results</a> | <a href="https://arxiv.org/abs/2108.08787">Paper</a> <p> </h4>Introduction
Mr. TyDi is a multi-lingual benchmark dataset built on TyDi, covering eleven typologically diverse languages. It is designed for mono-lingual retrieval, specifically to evaluate ranking with learned dense representations. Mr. TyDi is licensed under the Apache License 2.0.
Download
Version 1.1
1. Dataset (topic, qrels, folds, collections)
Ar | Bn | En | Fi | Id | Ja | Ko | Ru | Sw | Te | Th
The dataset (v1.1) is also available on HuggingFace Dataset:
2. Pre-built sparse index (for BM25)
Ar | Bn | En | Fi | Id | Ja | Ko | Ru | Sw | Te | Th
3. Pre-built dense index (for mDPR)
Ar | Bn | En | Fi | Id | Ja | Ko | Ru | Sw | Te | Th
4. Checkpoints
<details> <summary> <b> Previous Versions </b></summary>Version 1.0
1. Dataset
Ar | Bn | En | Fi | Id | Ja | Ko | Ru | Sw | Te | Th
2. Pre-built sparse index (for BM25)
Ar | Bn | En | Fi | Id | Ja | Ko | Ru | Sw | Te | Th
3. Pre-built dense index (for mDPR)
Ar | Bn | En | Fi | Id | Ja | Ko | Ru | Sw | Te | Th
</details>Baselines and Evaluation
The one-command reproduction (on v1.1) would require the recent dev version of Pyserini. Please follow this guidance to setup a dev installation for Pyserini.
This page only covers the scripts to reproduce searching. The indexes are all handled within Pyserini. That is, you won't need to manually download the above indexes or models to run the following scripts. For the scripts to reproduce the sparse and dense indexes, please refer to the Pyserini documentations:
Model | Documentation Link |
---|---|
Sparse Index | Ar \ |
Dense Index | Ar \ |
1. BM25
lang=arabic # one of {'arabic', 'bengali', 'english', 'finnish', 'indonesian', 'japanese', 'korean', 'russian', 'swahili', 'telugu', 'thai'}
lang_abbr=ar # one of {'ar', 'bn', 'en', 'fi', 'id', 'ja', 'ko', 'ru', 'sw', 'te', 'th'}
set_name=test # one of {'train', 'dev', 'test'}
runfile=runs/run.bm25.mrtydi-v1.1-${lang}.${set_name}.txt
python -m pyserini.search --bm25 \
--language ${lang_abbr} \
--topics mrtydi-v1.1-${lang}-${set_name} \
--index mrtydi-v1.1-${lang} \
--output ${runfile}
2. mDPR
lang=arabic # one of {'arabic', 'bengali', 'english', 'finnish', 'indonesian', 'japanese', 'korean', 'russian', 'swahili', 'telugu', 'thai'}
set_name=test # one of {'train', 'dev', 'test'}
runfile=runs/run.mdpr.mrtydi-v1.1-$lang.${set_name}.txt
python -m pyserini.dsearch \
--topics mrtydi-v1.1-${lang}-${set_name} \
--index mrtydi-v1.1-${lang}-mdpr-nq \
--encoder castorini/mdpr-question-nq \
--batch-size 36 \
--threads 12 \
--output
3. BM25+mDPR hybrid
To use the pre-set best alpha for each language:
lang=arabic # one of {'arabic', 'bengali', 'english', 'finnish', 'indonesian', 'japanese', 'korean', 'russian', 'swahili', 'telugu', 'thai'}
python scripts/hybrid.py --lang ${lang} \
--sparse ${bm25_runfile} \
--dense ${dense_runfile} \
--output ${runfile} \
--weight-on-dense \
--normalization
Or to run hybrid with any alpha
:
alpha=0.5
python scripts/hybrid.py --alpha ${alpha} \
--sparse ${bm25_runfile} \
--dense ${dense_runfile} \
--output ${runfile} \
--weight-on-dense \
--normalization
where the bm25_runfile
and dense_runfile
are prepared from the previous two steps.
4. Evaluate
python -m pyserini.eval.trec_eval -c -mrecip_rank -mrecall.100 ${qrels} ${runfile}
Results
Here we present the MRR@100 and Recall@100 scores after fixing a bug relavant to using multi-lingual models in Pyserini 0.13.0. The sparse scores are unaffected, whereas the mDPR and Hybrid scores are higher than the original reported ones to various degree. We also put the updated figures under the figures/ directory.
MRR@100
Ar | Bn | En | Fi | In | Ja | Ko | Ru | Sw | Te | Th | avg | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
BM25 (default) | 0.368 | 0.418 | 0.140 | 0.284 | 0.376 | 0.211 | 0.285 | 0.313 | 0.389 | 0.343 | 0.401 | 0.321 |
BM25 (tuned) | 0.366 | 0.413 | 0.150 | 0.287 | 0.382 | 0.217 | 0.280 | 0.329 | 0.396 | 0.424 | 0.416 | 0.333 |
mDPR (NQ) | 0.291 | 0.291 | 0.291 | 0.206 | 0.271 | 0.213 | 0.235 | 0.283 | 0.189 | 0.111 | 0.172 | 0.226 |
Hybrid | 0.500 | 0.555 | 0.328 | 0.377 | 0.481 | 0.360 | 0.361 | 0.455 | 0.415 | 0.418 | 0.507 | 0.426 |
Recall@100
Ar | Bn | En | Fi | In | Ja | Ko | Ru | Sw | Te | Th | avg | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
BM25 (default) | 0.793 | 0.869 | 0.537 | 0.719 | 0.843 | 0.645 | 0.619 | 0.648 | 0.764 | 0.758 | 0.853 | 0.732 |
BM25 (tuned) | 0.800 | 0.874 | 0.551 | 0.725 | 0.846 | 0.656 | 0.797 | 0.660 | 0.764 | 0.813 | 0.853 | 0.758 |
mDPR (NQ) | 0.650 | 0.779 | 0.678 | 0.568 | 0.685 | 0.584 | 0.533 | 0.647 | 0.528 | 0.366 | 0.515 | 0.594 |
Hybrid | 0.871 | 0.946 | 0.793 | 0.827 | 0.900 | 0.794 | 0.718 | 0.815 | 0.808 | 0.823 | 0.883 | 0.834 |
Citation
If you find our paper useful or use the dataset in your work, please cite our paper and the TyDi paper:
@article{mrtydi,
title={{Mr. TyDi}: A Multi-lingual Benchmark for Dense Retrieval},
author={Xinyu Zhang and Xueguang Ma and Peng Shi and Jimmy Lin},
year={2021},
journal={arXiv:2108.08787},
}
@article{tydiqa,
title={{TyDi QA}: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages},
author={Jonathan H. Clark and Eunsol Choi and Michael Collins and Dan Garrette and Tom Kwiatkowski and Vitaly Nikolaev and Jennimaria Palomaki}
year={2020},
journal={Transactions of the Association for Computational Linguistics}
}
Contact us
If you have any question or suggestions regarding the dataset, code or publication, please contact Xinyu Zhang (x978zhan[at]uwaterloo.ca)