Home

Awesome

TInfES

Type Inference Evaluation Scripts & Accessory Apps (used for the StaTIX benchmarking)

<!-- \authors: (c) Artem Lutov <artem@exascale.info>, [Soheil Roshankish](http://unibe-ch2.academia.edu/SoheilRoshankish/CurriculumVitae) \organization: [eXascale Infolab](http://exascale.info/) NOTE: the authors are temporaly hided for the tripple-blind conference review -->

\license: Apache License, Version 2.0

Content

Overview

Scripts and accessory Java applications used for the type inference benchmarking of StaTIX.

Scripts

Algorithms

Requirements

The scripts require any POSIX compatible execution environment (Linux or Unix). Java apps were tested on Java 1.8+.

Usage

To perform batch execution of the clustering algorithms copy / unpack the required algorithm to the dedicated directory <algdir> and link / copy there scripts/execfile.sh, scripts/<algname>.exs, exectime. Update paths in the <algname>.exs if required.
Run the batch execution: ./execfile.sh <algname>.exs. It produces evals_<algname>.rcp containing execution time and memory consumption measurements in the current directory and resulting clusterings (type inference) according to the specified parameters in the <algname>.exs.
Example (run StaTIX in the background):

$ nohup ./execfile.sh statix_m_rm.exs &>> statix_m_rm.log &

To perform batch evaluation of the type inference accuracy, create a dedicated directory <evals>/ and link / copy there scripts/evalgt.sh, scripts/linkfiles.sh, scripts/mkevaldirs.sh, xmeasures, gecmi and ground-truth files from the data/<datasets>_gt (or produce them). The ground-truth files contain for each #type property space separated subject ids, sequentially enumerated from 0. Run ./mkevaldirs.sh to create directories that will hold type inference results to be evaluated against the respective ground-truth files. Then run linkfiles.sh <results_dir> to link algorithm(s) results to the corresponding directories to be evaluated. And finally run the batch evaluations using evalgt.sh script. See evalgt.sh -h for details. It produces the required evaluations (eval_<evalapp-params>.txt files) in the current directory, which are the accuracy results of the type inference.
Example:

# Execute the algorithm(s) to infer the types (clusters)
$ nohup ./execfile.sh statix_m_rm_j_w.exs &>> statix_m_rm_j_w.log &
# Rename and remove directories from the previous mapping if any
$ ./renamedirs.sh _tmp && rm -r *_tmp/
# Make evaluation directories and link the files from the type inference results
$ ./mkevaldirs.sh
$ ./linkfiles.sh ../results/statix/kenzabased/
$ ./linkfiles.sh ../results/statix/biomedical/
$ ./linkfiles.sh ../results/statix/opengov/
# Evaluate results by the required measures
$ ./evalgt.sh ./xmeasures -fh 1 museum soccerplayer country politician film mixen gendr-stat lsr-stat gendr_gene_expression wikipathways-stat genage_human lsr libraries bauhist-fotosamm schools hist_munic_reg &> evals_fh.log
$ nohup ./evalgt.sh ./gecmi 3 museum soccerplayer country politician film mixen gendr-stat lsr-stat gendr_gene_expression wikipathways-stat genage_human lsr libraries bauhist-fotosamm schools hist_munic_reg &> evals_nmi-gecmi.log &

Benchmarking Results

StaTIX in the non-supervised mode with the following parameters:

Accuracy

Harmonic F1-Score (higher is better)

F1h

F1-Measures, Precision and Recall for the labeled types (higher is better)

\ AlgorithmStaTIXStaTIX-rm\StaTIX-rm-m[-f]SDASDType
Dataset \F1F1F1PRF1PRF1PR
museum0.8660.8660.8661.0000.7630.5390.3800.9270.2090.1200.785
soccerplayer0.7890.7890.7891.0000.6520.6950.5740.8820.4470.3390.657
country0.8400.8400.8401.0000.7250.6320.4780.9300.2490.1550.634
politician0.7320.7320.7560.9820.6150.7040.5900.8740.4710.4030.568
film1.0001.0001.0001.0001.0000.8390.7221.0000.4350.2781.000
mixen0.5050.7230.7510.8690.6620.5590.4120.8730.3780.3600.398
gendrgene0.8060.8060.8060.7570.8610.8890.9870.809
lsr0.9120.9900.9901.0000.9810.9980.9960.999
bauhist1.0001.0001.0001.0001.0001.0001.0001.000
schools1.0001.0001.0001.0001.0001.0001.0001.000
histmunic0.9500.9500.9581.0000.920

Efficiency

Execution Time (lower is better)

Execution Time

Memory Consumption (RAM) (lower is better)

RAM

Note: Please, star this project if you use it.

Related Projects