Home

Awesome

Evaluating Gender Bias in Machine Translation

This repo contains code and data for reproducing the experiments in Evaluating Gender Bias in Machine Translation Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer, (ACL 2019), and Gender Coreference and Bias Evaluation at WMT 2020, Tom Kocmi, Tomasz Limisiewicz, Gabriel Stanovsky (WMT2020).

Citing

@InProceedings{Stanovsky2019ACL,
  author    = {Gabriel Stanovsky and Noah A. Smith and Luke Zettlemoyer},
  title     = {Evaluating Gender Bias in Machine Translation},
  booktitle = {ACL},
  month     = {June},
  year      = {2019},
  address   = {Florence, Italy},
  publisher = {Association for Computational Linguistics}
}

Requirements

Evaluation in Polish

Install

./install.sh

Running our experiments

This is the entry point for all our experiments: scripts/evaluate_all_languages.sh. Run all of the following from the src folder. Output logs will be written to the given path.

Adding an MT system

  1. Translate the file in data/aggregates/en.txt to the languages in our evaluation method.
  2. Put the transalations in translations/your-mt-system/en-targetLanguage.txt where each sentence is in a new line, which has the following format original-sentence ||| translated sentence. See this file for an example.
  3. Add your translator in the mt_systems enumeration in the evaluation script.