Awesome

XAlign: Cross-lingual-Fact-to-Text-Alignment-and-Generation-for-Low-Resource-Languages

In this work, we propose the creation of cross-lingual fact-to-text dataset, XAlign accepted at WebConf-2022 poster-demo track. It consist of English WikiData triples/facts mapped to sentences from low resources Wikipedia.

We explored two different unsupervised methods to solve cross-lingual alignment task based on:

• Transfer learning from NLI
• Distant supervision from another English-only dataset

This repository consists of steps for executing the cross-lingual alignment approaches and finetuning mT5 for data-to-text generation on XAlign. One can find more details, analyses, and baseline results in our paper.

Installation

Install the required packgaes as follow:

pip install -r requirements.txt

Dataset

Dataset releases

• v2.0 (Sep 2022): Extended to four additional languages: Punjabi (pa), Malayalam (ml), Assamese (as) and Oriya (or).
• v1.0 (Apr 2022): Introduced the cross-lingual data-to-text into 8 langauges: Hindi (hi), Marathi (mr), Gujarati (gu), Telugu (te), Tamil (ta), Kannada (kn), Bengali (bn), and monolingual dataset in English (en).

Data Fields

Each record consist of the following entries:

• sentence (string) : Native language wikipedia sentence. (non-native language strings were removed.)
• facts (List[Dict]) : List of facts associated with the sentence where each fact is stored as dictionary.
• language (string) : Language identifier.

The facts key contains list of facts where each facts is stored as dictionary. A single record within fact list contains following entries:

• subject (string) : central entity.
• object (string) : entity or a piece of information about the subject.
• predicate (string) : relationship that connects the subject and the object.
• qualifiers (List[Dict]) : It provide additional information about the fact, is stored as list of qualifier where each record is a dictionary. The dictionary contains two keys: qualifier_predicate to represent property of qualifer and qualifier_object to store value for the qualifier's predicate.

Data Instances

Example from English

{
  "sentence": "Mark Paul Briers (born 21 April 1968) is a former English cricketer.",
  "facts": [
    {
      "subject": "Mark Briers",
      "predicate": "date of birth",
      "object": "21 April 1968",
      "qualifiers": []
    },
    {
      "subject": "Mark Briers",
      "predicate": "occupation",
      "object": "cricketer",
      "qualifiers": []
    },
    {
      "subject": "Mark Briers",
      "predicate": "country of citizenship",
      "object": "United Kingdom",
      "qualifiers": []
    }
  ],
  "language": "en"
}

Example from one of the low-resource languages (i.e. Hindi)

{
  "sentence": "बोरिस पास्तेरनाक १९५८ में साहित्य के क्षेत्र में नोबेल पुरस्कार विजेता रहे हैं।",
  "facts": [
    {
      "subject": "Boris Pasternak",
      "predicate": "nominated for",
      "object": "Nobel Prize in Literature",
      "qualifiers": [
        {
          "qualifier_predicate": "point in time",
          "qualifier_object": "1958"
        }
      ]
    }
  ],
  "language": "hi"
}

Gold standard Test dataset

We manually annotated the test dataset across 8 languages with the help of crowd-sourced annotators.

Train and validation dataset (automatically aligned)

We have automatically created a large collection of well aligned sentence-fact pair across languages using the best cross-lingual aligner evaluated on gold standard test datasets.

Cross-lingual Alignment Approaches

1) Transfer learning from NLI

Before executing the code, download the XNLI dataset from here.

To execute the mT5 based approach, follow the steps:

$ cd XNLI-based-models/finetune_mt5

Copy xnli_dataset.zip (downloaded before) to ./datasets and unzip. Finally execute the command:

$ python main.py --epochs 5 --gpus 1 --batch_size 16 --max_seq_len 200 --learning_rate 1e-3 --model_name google/mt5-large --fp16 0

To execute the MuRIL or XLM-RoBERTa based approaches, follow the steps:

$ cd XNLI-based-models/finetune_multilingual_encoder_models

Copy xnli_dataset.zip (downloaded before) to ./datasets and unzip.Finally execute the command:

$ python main.py --epochs 5 --gpus 1 --batch_size 32 --max_seq_len 200 --learning_rate <lr> --model_name <model_name> --fp16 1

where,

• model_name can be 'google/muril-large-cased' or 'xlm-roberta-large'
• learning_rate must be '1e-5' for 'xlm-roberta-large' or '2e-5' for 'google/muril-large-cased'

2) Distant supervision using KELM dataset

Before executing the code, download the multi-lingual KELM dataset from here.

To execute the mT5 based approach, follow the steps:

$ cd distant_supervision/finetune_mt5

Copy multilingual-KELM-dataset.zip (downloaded before) to ./datasets directory and unzip. Finally execute the command:

$ python main.py --epochs 5 --gpus 1 --batch_size 16 --max_seq_len 200 --learning_rate 1e-3 --model_name google/mt5-large --fp16 0

To execute the MuRIL or XLM-RoBERTa based approaches, follow the steps:

$ cd distant_supervision/finetune_multilingual_encoder_models

Copy multilingual-KELM-dataset.zip (downloaded before) to ./datasets directory and unzip. Finally execute the command:

$ python main.py --epochs 5 --gpus 1 --batch_size 32 --max_seq_len 200 --learning_rate <lr> --model_name <model_name> --fp16 1

where,

• model_name can be 'google/muril-large-cased' or 'xlm-roberta-large'
• learning_rate must be '1e-5' for 'xlm-roberta-large' or '2e-5' for 'google/muril-large-cased'

Cross-lingual Alignment Results

Following are the F1-score for cross-lingual alignment over gold standard test datasets.

Cross-lingual Data-to-Text Generation

Before procedding, copy XAlign-dataset.zip (available upon request) to data-to-text-generator/mT5-baseline/datasets folder and unzip.

To finetune the best baseline on the XAlign, follow the steps:

$ cd data-to-text-generator/mT5-baseline
$ python main.py --epochs 30 --gpus 1 --batch_size 2 --src_max_seq_len 250 --tgt_max_seq_len 200 --learning_rate 1e-3 --model_name google/mt5-small --online_mode 0 --use_pretrained 1 --lang hi,mr,te,ta,en,gu,bn,kn --verbose --enable_script_unification 1 

To evaluate the trained model, follow the steps:

$ cd data-to-text-generator/mT5-baseline
$ python main.py --epochs 30 --gpus 1 --batch_size 4 --src_max_seq_len 250 --tgt_max_seq_len 200 --learning_rate 1e-3 --model_name google/mt5-small --online_mode 0 --use_pretrained 1 --lang hi,mr,te,ta,en,gu,bn,kn enable_script_unification 1 --inference

Cross-lingual Data-to-Text Generation Results

BLEU obtained on the Test dataset on XAlign.

Contributors

• Tushar Abhishek
• Shivprasad Sagare
• Bhavyajeet Singh
• Anubhav Sharma
• Manish Gupta
• Vasudeva Varma

Citation

One can cite our paper as follows:

@article{abhishek2022xalign,
  title={XAlign: Cross-lingual Fact-to-Text Alignment and Generation for Low-Resource Languages},
  author={Abhishek, Tushar and Sagare, Shivprasad and Singh, Bhavyajeet and Sharma, Anubhav and Gupta, Manish and Varma, Vasudeva},
  journal={arXiv preprint arXiv:2202.00291},
  year={2022}
}