Awesome

MT@BigScience

Evaluation results for Machine Translation within the BigScience project. Evaluation is carried out using the BigScience fork of lm-evaluation-harness coupled with the eval-hackathon branch of PromptSource. N.B. Updates of latest versions are currently ongoing and will be available shortly.

Citation

This repository contains codes and outputs to accompany the paper "Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM". Please cite the following:

@inproceedings{bawden-yvon-bloom-mt-2023,
    author = {Bawden, Rachel and Yvon, François},
    title = {Investigating the Translation Performance of a Large Multilingual
Language Model: the Case of {BLOOM}},
    booktitle = {Proceedings of the 24th Annual Conference of the European Association for Machine Translation},
    url = {https://arxiv.org/abs/2303.01911},
    year = {2023},
    notes = {To appear}
}

Outputs and evaluation

Extract all predictions and evaluate

python scripts/process_results_{flores,diabla,wmt}.py

This extracts all predictions from .jsonl files into .tsv files and calculates BLEU and COMET scores. These are written out to the following folders:

• outputs/{wmt14_hi_en,wmt14_fr_en,flores-101}/{0,1,2,5}-shot/{comet,bleu}-results.tsv for WMT and flores-101
• outputs/diabla/{0,1}-shot/{comet,bleu}-results.{English-French,French-English}.tsv for DiaBLa.

Three versions of each output are generated:

1. The original outputs
2. The outputs truncated at the first newline (newline-cut)
3. The outputs truncated at the first newline or before the first repetition of the 'xglm' prompt (newline-cut-custom-truncate). This corresponds to the truncated outputs from the paper.

Generate latex tables:

python scripts/make-tables-{flores,diabla,wmt}.py

Results (BLEU scores)

Cross-dataset and model comparison (focus on English-French and English-Hindi)

WMT14 results (Original outputs)

DiaBLa results (Original outputs)

Flores-101 results (Original outputs)

WMT14 results (Truncated outputs)

DiaBLa results (Truncated outputs)

Flores-101 results (Truncated outputs)

Flores-101: High-resource, 1-shot

(Original outputs with no postprocessing)

Flores-101:High-mid resource, 1-shot

(Original outputs with no postprocessing)

Flores-101: Low-resource, 1-shot

(Original outputs with no postprocessing)

Flores-101: Romance languages, 1-shot

(Original outputs with no postprocessing)

Flores-101: Bengali→English MT, Transfer using 1-shot example from a different language direction

(Original outputs with no postprocessing)

DiaBLa context results (1-shot with differing source of context)

The 1-shot example can be:

• from either anywhere in the document (Origin=Rand.) or the previous sentence (Origin=Prev.)
• from any language direction (en→fr or fr→en) regardless of the language direction of the current example (Direction=rand.), from the same direction as the current example (Direction=same) or the opposite direction (Direction=opposite).