Awesome
Files2ROUGE
Motivations
Given two files with the same number of lines, files2rouge
calculates the average ROUGE scores of each sequence (=line). Each sequence may contain multiple sentences. In this case, the end of sentence string must be passed using the --eos
flag (default: "."). Running files2rouge
with a wrong eos delimiter may lead to incorrect ROUGE-L score.
You may also be interested in a Python implementation (instead of a wrapper): https://github.com/pltrdy/rouge.
$ files2rouge --help
usage: files2rouge [-h] [-v] [-a ARGS] [-s SAVETO] [-e EOS] [-m] [-i]
reference summary
Calculating ROUGE score between two files (line-by-line)
positional arguments:
reference Path of references file
summary Path of summary file
optional arguments:
-h, --help show this help message and exit
-v, --verbose Prints ROUGE logs
-a ARGS, --args ARGS ROUGE Arguments
-s SAVETO, --saveto SAVETO
File to save scores
-e EOS, --eos EOS End of sentence separator (for multisentence).
Default: "."
-m, --stemming DEPRECATED: stemming is now default behavior
-nm, --no_stemming Switch off stemming
-i, --ignore_empty
Getting Started
0) Install prerequisites
pip install -U git+https://github.com/pltrdy/pyrouge
(NOTE: running pip install pyrouge
would not work as the package is out of date on PyPI)
1) Clone the repo, setup the module and ROUGE
git clone https://github.com/pltrdy/files2rouge.git
cd files2rouge
python setup_rouge.py
python setup.py install
Do not forget to run setup_rouge
2) Run files2rouge.py
files2rouge references.txt summaries.txt
Outputs:
When --verbose
is set, the script prints progress and remaining time on stderr
. This can be changed using --verbose
in order to outputs ROUGE execution logs.
Default output example:
Preparing documents...
Running ROUGE...
---------------------------------------------
1 ROUGE-1 Average_R: 0.28242 (95%-conf.int. 0.25721 - 0.30877)
1 ROUGE-1 Average_P: 0.30157 (95%-conf.int. 0.27114 - 0.33506)
1 ROUGE-1 Average_F: 0.28196 (95%-conf.int. 0.25704 - 0.30722)
---------------------------------------------
1 ROUGE-2 Average_R: 0.10395 (95%-conf.int. 0.08298 - 0.12600)
1 ROUGE-2 Average_P: 0.11458 (95%-conf.int. 0.08873 - 0.14023)
1 ROUGE-2 Average_F: 0.10489 (95%-conf.int. 0.08303 - 0.12741)
---------------------------------------------
1 ROUGE-L Average_R: 0.25231 (95%-conf.int. 0.22709 - 0.27771)
1 ROUGE-L Average_P: 0.26830 (95%-conf.int. 0.23834 - 0.29818)
1 ROUGE-L Average_F: 0.25142 (95%-conf.int. 0.22741 - 0.27533)
Elapsed time: 0.458 secondes
Call files2rouge
from Python
import files2rouge
files2rouge.run(hyp_path, ref_path)
ROUGE Args
One can specify which ROUGE args to use using the flag --args
(or -a
).
The default behavior is equivalent to:
files2rouge reference.txt summary.txt -a "-c 95 -r 1000 -n 2 -a" # be sure to write args betwen double-quotes
You can find more informations about these arguments here
Known issues
ROUGE-1.5.5.pl - XML::Parser dependency error
: see issue #9.
More informations
- ROUGE Original Paper (Lin 2004)
- ROUGE-1.5.5/README.txt
- Use cases:
- About
files2rouge.py
: runfiles2rouge.py --help