Awesome
JPMML-SparkML ![Build Status](https://github.com/jpmml/jpmml-sparkml/workflows/maven/badge.svg)
Java library and command-line application for converting Apache Spark ML pipelines to PMML.
Table of Contents
Features
Overview
- Functionality:
- Thorough collection, analysis and encoding of feature information:
- Names.
- Data and operational types.
- Valid, invalid and missing value spaces.
- Pipeline extensions:
- Pruning.
- Model verification.
- Conversion options.
- Thorough collection, analysis and encoding of feature information:
- Extensibility:
- Rich Java APIs for developing custom converters.
- Automatic discovery and registration of custom converters based on
META-INF/sparkml2pmml.properties
resource files. - Direct interfacing with other JPMML conversion libraries such as JPMML-LightGBM and JPMML-XGBoost.
- Production quality:
- Complete test coverage.
- Fully compliant with the JPMML-Evaluator library.
Supported libraries
<details> <summary>Apache Spark ML</summary>Examples: main.py
- Feature extractors, transformers and selectors:
feature.Binarizer
feature.Bucketizer
feature.ChiSqSelectorModel
(the result of fitting afeature.ChiSqSelector
)feature.ColumnPruner
feature.CountVectorizerModel
(the result of fitting afeature.CountVectorizer
)feature.IDFModel
(the result of fitting afeature.IDF
)feature.ImputerModel
(the result of fitting afeature.Imputer
)feature.IndexToString
feature.Interaction
feature.MaxAbsScalerModel
(the result of fitting afeature.MaxAbsScaler
)feature.MinMaxScalerModel
(the result of fitting afeature.MinMaxScaler
)feature.NGram
feature.OneHotEncoderModel
(the result of fitting afeature.OneHotEncoder
)feature.PCAModel
(the result of fitting afeature.PCA
)feature.QuantileDiscretizer
feature.RegexTokenizer
feature.RFormulaModel
(the result of fitting afeature.RFormula
)feature.SQLTransformer
- Subqueries.
- Control flow expressions
case when
andif
. - Arithmetic operators
+
,-
,*
and/
. - Comparison operators
<
,<=
,==
,>=
and>
. - Logical operators
and
,or
andnot
. - Math functions
abs
,ceil
,exp
,expm1
,floor
,hypot
,ln
,log10
,log1p
,pow
andrint
. - Trigonometric functions
sin
,asin
,sinh
,cos
,acos
,cosh
,tan
,atan
,tanh
. - Aggregation functions
greatest
andleast
. - RegExp functions
regexp_replace
andrlike
. - String functions
char_length
,character_length
,concat
,lcase
,length
,lower
,replace
,substring
,trim
,ucase
andupper
. - Type cast functions
boolean
,cast
,double
,int
andstring
. - Value functions
in
,isnan
,isnull
,isnotnull
,negative
andpositive
.
feature.StandardScalerModel
(the result of fitting afeature.StandardScaler
)feature.StopWordsRemover
feature.StringIndexerModel
(the result of fitting afeature.StringIndexer
)feature.Tokenizer
feature.VectorAssembler
feature.VectorAttributeRewriter
feature.VectorIndexerModel
(the result of fitting afeature.VectorIndexer
)feature.VectorSizeHint
feature.VectorSlicer
- Prediction models:
classification.DecisionTreeClassificationModel
classification.GBTClassificationModel
classification.LinearSVCModel
classification.LogisticRegressionModel
classification.MultilayerPerceptronClassificationModel
classification.NaiveBayesModel
classification.RandomForestClassificationModel
clustering.KMeansModel
fpm.FPGrowthModel
regression.DecisionTreeRegressionModel
regression.GBTRegressionModel
regression.GeneralizedLinearRegressionModel
regression.LinearRegressionModel
regression.RandomForestRegressionModel
- Prediction model chains:
PipelineModel
- Referencing the prediction column (
HasPredictionCol#getPredictionCol()
) of earlier clustering, classification and regression models. - Referencing the predicted probabilities column (
HasProbabilityCol#getProbabilityCol()
) of earlier classification models.
- Hyperparameter selectors and tuners:
- Feature transformers:
org.jpmml.sparkml.feature.InvalidCategoryTransformer
org.jpmml.sparkml.feature.SparseToDenseTransformer
Examples: LightGBMAuditNA.scala, LightGBMAutoNA.scaka, etc.
- Prediction models:
Examples: XGBoostAuditNA.scala, XGBoostAutoNA.scala, etc.
- Prediction models:
Prerequisites
- Apache Spark 3.0.X, 3.1.X, 3.2.X, 3.3.X, 3.4.X or 3.5.X.
Installation
Library
JPMML-SparkML library JAR file (together with accompanying Java source and Javadocs JAR files) is released via Maven Central Repository.
The current version is 2.5.1 (20 June, 2024).
<dependency>
<groupId>org.jpmml</groupId>
<artifactId>pmml-sparkml</artifactId>
<version>2.5.1</version>
</dependency>
Compatibility matrix
Active development branches:
Apache Spark version | JPMML-SparkML branch |
---|---|
3.0.X | 2.0.X |
3.1.X | 2.1.X |
3.2.X | 2.2.X |
3.3.X | 2.3.X |
3.4.X | 2.4.X |
3.5.X | master |
Archived development branches:
Apache Spark version | JPMML-SparkML branch |
---|---|
1.5.X and 1.6.X | 1.0.X |
2.0.X | 1.1.X |
2.1.X | 1.2.X |
2.2.X | 1.3.X |
2.3.X | 1.4.X |
2.4.X | 1.5.X |
1.6.X | |
1.7.X | |
1.8.X |
Example application
Enter the project root directory and build using Apache Maven:
mvn clean install
The build produces two JAR files:
pmml-sparkml/target/pmml-sparkml-2.5-SNAPSHOT.jar
- Library JAR file.pmml-sparkml-exampletarget/pmml-sparkml-example-executable-2.5-SNAPSHOT.jar
- Example application JAR file.
Usage
Library
Fitting a Spark ML pipeline:
import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.classification.DecisionTreeClassifier
import org.apache.spark.ml.feature.RFormula
val irisData = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("Iris.csv")
val irisSchema = irisData.schema
val rFormula = new RFormula().setFormula("Species ~ .")
val dtClassifier = new DecisionTreeClassifier().setLabelCol(rFormula.getLabelCol).setFeaturesCol(rFormula.getFeaturesCol)
val pipeline = new Pipeline().setStages(Array(rFormula, dtClassifier))
val pipelineModel = pipeline.fit(irisData)
Converting the fitted Spark ML pipeline to an in-memory PMML class model object:
import org.jpmml.sparkml.PMMLBuilder
val pmml = new PMMLBuilder(irisSchema, pipelineModel).build()
The representation of individual Spark ML pipeline stages can be customized via conversion options:
import org.jpmml.sparkml.PMMLBuilder
import org.jpmml.sparkml.model.HasTreeOptions
val dtClassifierModel = pipelineModel.stages(1)
val pmml = new PMMLBuilder(irisSchema, pipelineModel).putOption(dtClassifierModel, HasTreeOptions.OPTION_COMPACT, false).putOption(dtClassifierModel, HasTreeOptions.OPTION_ESTIMATE_FEATURE_IMPORTANCES, true).build()
Viewing the in-memory PMML class model object:
import javax.xml.transform.stream.StreamResult
import org.jpmml.model.JAXBUtil
JAXBUtil.marshalPMML(pmml, new StreamResult(System.out))
Example application
The example application JAR file contains an executable class org.jpmml.sparkml.example.Main
, which can be used to convert a pair of serialized org.apache.spark.sql.types.StructType
and org.apache.spark.ml.PipelineModel
objects to PMML.
The example application JAR file does not include Apache Spark runtime libraries. Therefore, this executable class must be executed using Apache Spark's spark-submit
helper script.
For example, converting a pair of Spark ML schema and pipeline serialization files pmml-sparkml/src/test/resources/schema/Iris.json
and pmml-sparkml/src/test/resources/pipeline/DecisionTreeIris.zip
, respectively, to a PMML file DecisionTreeIris.pmml
:
spark-submit --master local --class org.jpmml.sparkml.example.Main pmml-sparkml-example/target/pmml-sparkml-example-executable-2.5-SNAPSHOT.jar --schema-input pmml-sparkml/src/test/resources/schema/Iris.json --pipeline-input pmml-sparkml/src/test/resources/pipeline/DecisionTreeIris.zip --pmml-output DecisionTreeIris.pmml
Getting help:
spark-submit --master local --class org.jpmml.sparkml.example.Main pmml-sparkml-example/target/pmml-sparkml-example-executable-2.5-SNAPSHOT.jar --help
Documentation
- Training PySpark LightGBM pipelines
- Converting logistic regression models to PMML documents
- Deploying Apache Spark ML pipeline models on Openscoring REST web service
- Converting Apache Spark ML pipeline models to PMML documents
License
JPMML-SparkML is licensed under the terms and conditions of the GNU Affero General Public License, Version 3.0.
If you would like to use JPMML-SparkML in a proprietary software project, then it is possible to enter into a licensing agreement which makes JPMML-SparkML available under the terms and conditions of the BSD 3-Clause License instead.
Additional information
JPMML-SparkML is developed and maintained by Openscoring Ltd, Estonia.
Interested in using Java PMML API software in your company? Please contact info@openscoring.io