Home

Awesome

OpenMEVA

Contributed by Jian Guan, Zhexin Zhang. Thank Jiaxin Wen for DeBugging.

OpenMEVA is a benchmark for evaluating open-ended story generation metrics (Please refer to the Paper List for more information about Open-eNded Language Generation tasks) described in the paper: OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics (ACL 2021 Long Paper). Besides, OpenMEVA also provides an open-source and extensible toolkit for metric implementation, evaluation, comparison, and analysis, as well as data perturbation techniques to help generate large numbers of customized test cases. We expect the toolkit to empower fast development of automatic metrics.

🚀 NEWS: THE OPENMEVA BENCHMARK ARE AVAILABLE AT 🤗 HuggingFace NOW.

Contents

Introduction for Language Generation Evaluation

Since human evaluation is time-consuming, expensive, and difficult to reproduce, the community commonly uses automatic metrics for evaluation. We roughly divide existing metrics as follows:

The existing generation models are still far from human ability to generate reasonable texts, particularly for open-ended language generation tasks such as story generation. One important factor that hinders the research is the lack of powerful metrics for measuring generation quality. Therefore, we propose OpenMEVA as the standard paradigm for measuring progress of metrics.

Install

Clone the repository from our github page (don't forget to star us!)

git clone https://github.com/thu-coai/OpenMEVA.git

Then install all the requirements:

pip install -r requirements.txt

Then install the package with

python setup.py install

If you also want to modify the code, run this:

python setup.py develop

Toolkit

I. Metrics Interface

1. Metric List

We publish the standard implementation for the following metrics:

2. Usage

It is handy to construct a metric object and use it to evaluate given examples:

from eva.bleu import BLEU
metric = BLEU()

# for more information about the metric
print(metric.info)

# data is a list of dictionary [{"context": ..., "candidate":..., "reference": ...}]
print(metric.compute(data))

We present a python file test.py as an instruction to access the API.

These metrics are not exhaustive, it is a starting point for further metric research. We welcome any pull request for other metrics (requiring implementation of only three methods including __init__, info, compute).

3. Training Learnable Metrics

Execute the following command for training learnable metrics:

cd ./eva/model

# training language model for computing forward perplexity
bash ./run_language_modeling.sh

# training the unreferenced model for computing RUBER (RNN version)
bash ./run_ruber_unrefer.sh

# training the unreferenced model for computing RUBER (BERT version)
bash ./run_ruber_unrefer_bert.sh

# training the model for computing UNION
bash ./run_union.sh

II. Evaluating Human Scores

The python file test.py also includes detailed instruction to access the API for evaluating human scores.

1. Constructing

from eva.heva import Heva

# list of all possible human scores (int/float/str).
all_possible_score_list = [1,2,3,4,5]

# construct an object for following evaluation
heva = Heva(all_possible_score_list)

2. Consistency of human scores

# list of human score list, each row includes all the human scores for an example
human_score_list = [[1,3,2], [1,3,3], [2,3,1], ...]

print(heva.consistency(human_score_list))
# {"Fleiss's kappa": ..., "ICC correlation": ..., "Kendall-w":..., "krippendorff's alpha":...}
# the results includes correlation and p-value for significance test.

3. Mean Test for scores of examples from different source

# list of metric scores (float)
metric_score_1, metric_score_2 = [3.2, 2.4, 3.1,...], [3.5, 1.2, 2.3, ...]

# T-test for the means of two independent samples of scores.
print(heva.mean_test(metric_score_1, metric_score_2))
# {"t-statistic": ..., "p-value": ...}

4. Distribution of human scores

# list of human scores (float)
human_score = [2.0, 4.2, 1.2, 4.9, 2.6, 3.1, 4.0, 1.5,...]

# path for saving the figure of distribution
figure_path = "./figure"

# indicating the source of the annotated examples. default: ""
model_name = "gpt"

# plot the figure of distribution of human scores
heva.save_distribution_figure(score=human_score, save_path=figure_path, model_name=model_name, ymin=0, ymax=50)

5. Correlation between human and metric scores

# list of human scores (float)
human_score = [2.0, 4.2, 1.2, 4.9, 2.6, 3.1, 4.0, 1.5,...]

# list of metric scores (float)
metric_score = [3.2, 2.4, 3.1, 3.5, 1.2, 2.3, 3.5, 1.1,...]

# computing correlation
print(heva.correlation(metric_score, human_score))

# path for saving the figure of distribution
figure_path = "./figure"

# indicating the source of the metric scores. default: ""
metric_name = "bleu"

# plot the figure of metric score vs. human scores
heva.save_correlation_figure(human_score, metric_score, save_path=figure_path, metric_name=metric_name)

III. Perturbation Techniques

1. Perturbation List

We provide perturbation techniques in following aspects to create large scale test cases for evaluating comprehensive capabilities of metrics:

2. Usage

It is handy to construct a perturbation object and use it to perturb given examples:

from eva.perturb.perturb import *
custom_name = "story"
method = add_typos(custom_name)

# data is a list of dictionary [{"id":0, "ipt": ..., "truth":...}, ...]
print(method.construct(data))
# the perturbed examples can be found under the directory "custom_name"

We present a python file test_perturb.py as an instruction to access the API.

You can download dependent files for some perturbation techniques by executing the following command:

cd ./eva/perturb
bash ./download.sh

You can also download them by THUCloud or Google Drive.

These perturbation techniques are not exhaustive, it is a starting point for further evaluation research. We welcome any pull request for other perturbation techniques (requiring implementation of only two methods including __init__, construct).

Note:bookmark_tabs: We adopt uda for back translation. We provide an example eva/perturb/back_trans_data/story_bt.json to indicate the format of the back translation result. And you can download the results for ROCStories and WritingPrompts by THUCloud or Google Drive.

Benchmark

I. Datasets

1. Machine-Generated Stories (MAGS) with manual annotation

We provide annotated stories from ROCStories (ROC) and WritingPrompts (WP). Some statistics are as follows:

<img src="./figure/stat.png" style="zoom:30%;" />

Boxplot of annotation scores for each story source (Left: ROC, Right: WP):

2. Auto-Constructed Stories (ACTS)

We create large-scale test examples based on ROC and WP by aforementioned perturbation techniques. ACTS includes examples for different test types, i.e., discrimination test and invariance test.

3. Download & Data Instruction

You can download the whole dataset by THUCloud or Google Drive.

├── data
   └── `mags_data`
       ├── `mags_roc.json`	# sampled stories and corresponding human annotation.   
       ├── `mags_wp.json`		# sampled stories and corresponding human annotation.       
   └── `acts_data`
       ├── `roc`
              └── `roc_train_ipt.txt`	# input for training set
              └── `roc_train_opt.txt`	# output for training set
              └── `roc_valid_ipt.txt`	# input for validation set
              └── `roc_valid_opt.txt`	# output for validation set
              └── `roc_test_ipt.txt`	# input for test set
              └── `roc_test_opt.txt`	# output for test set
              └── `discrimination_test`                        
                 ├── `roc_lexical_rept.txt`
                 ├── `roc_lexical_rept_perturb.txt`										
                 ├── `roc_semantic_rept.txt`
                 ├── `roc_semantic_rept_perturb.txt`
                 ├── `roc_character.txt`
                 ├── `roc_character_perturb.txt`
                 ├── `roc_commonsense.txt`
                 ├── `roc_commonsense_perturb.txt`												
                 ├── `roc_coherence.txt`
                 ├── `roc_coherence_perturb.txt`
                 ├── `roc_consistency.txt`
                 ├── `roc_consistency_perturb.txt`								
                 ├── `roc_cause.txt`
                 ├── `roc_cause_perturb.txt`       										
                 ├── `roc_time.txt`
                 ├── `roc_time_perturb.txt`                    
              └── `invariance_test`
                 ├── `roc_synonym_substitute_perturb.txt`
                 ├── `roc_semantic_substitute_perturb.txt`
                 ├── `roc_contraction_perturb.txt`
                 ├── `roc_delete_punct_perturb.txt`
                 ├── `roc_typos_perturb.txt`
                 ├── `roc_negative_sample.txt`	# sampled negative samples from the discrimination test.	
                 ├── `roc_negative_sample_synonym_substitute_perturb.txt`
                 ├── `roc_negative_sample_semantic_substitute_perturb.txt`
                 ├── `roc_negative_sample_contraction_perturb.txt`
                 ├── `roc_negative_sample_delete_punct_perturb.txt`
                 ├── `roc_negative_sample_typos_perturb.txt`
       ├── `wp`
              └── ...

II. Tasks

OpenMEVA includes a suite of tasks to test comprehensive capabilities of metrics:

1. Correlation with human scores (based on MAGS)

<img src="./figure/task1.png" />

2. Generalization across generation models and dataset (for learnable metrics, based on MAGS)

<img src="./figure/task2.png" style="zoom:30%;" />

3. Judgment in general linguistic features (based on the discrimination test set of ACTS)

4. Robustness to rationality-preserving perturbations (based on the invariance test set of ACTS)

Note: The smaller absolute value of correlation is the better.

5. Fast Test

You can test these capabilities of new metrics by following command:

cd ./benchmark

# test correlation with human scores and generalization
python ./corr_gen.py

# test judgment
python ./judge.py

# test robustness
python ./robust.py

We take BLEU and Forward Perplexity as examples in the python files. You can test your own metrics by minor modification.

How to Cite

@misc{guan2021openmeva,
      title={OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics}, 
      author={Jian Guan and Zhexin Zhang and Zhuoer Feng and Zitao Liu and Wenbiao Ding and Xiaoxi Mao and Changjie Fan and Minlie Huang},
      year={2021},
      eprint={2105.08920},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

It's our honor to help you better explore language generation evaluation with our toolkit and benchmark.