Home

Awesome

Hyperparameter Optimization of Machine Learning Algorithms

This code provides a hyper-parameter optimization implementation for machine learning algorithms, as described in the paper:
L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.

To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization.

This paper and code will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.

Paper

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
One-column version: arXiv
Two-column version: Elsevier

Quick Navigation

Section 3: Important hyper-parameters of common machine learning algorithms
Section 4: Hyper-parameter optimization techniques introduction
Section 5: How to choose optimization techniques for different machine learning models
Section 6: Common Python libraries/tools for hyper-parameter optimization
Section 7: Experimental results (sample code in "HPO_Regression.ipynb" and "HPO_Classification.ipynb")
Section 8: Open challenges and future research directions
Summary table for Sections 3-6: Table 2: A comprehensive overview of common ML models, their hyper-parameters, suitable optimization techniques, and available Python libraries
Summary table for Sections 8: Table 10: The open challenges and future directions of HPO research

Implementation

Sample code for hyper-parameter optimization implementation for machine learning algorithms is provided in this repository.

Sample code for Regression problems

HPO_Regression.ipynb
Dataset used: Boston-Housing

Sample code for Classification problems

HPO_Classification.ipynb
Dataset used: MNIST

Machine Learning & Deep Learning Algorithms

Hyperparameter Configuration Space

ML ModelHyper-parameterTypeSearch Space
RF Classifiern_estimatorsDiscrete[10,100]
max_depthDiscrete[5,50]
min_samples_splitDiscrete[2,11]
min_samples_leafDiscrete[1,11]
criterionCategorical'gini', 'entropy'
max_featuresDiscrete[1,64]
SVM ClassifierCContinuous[0.1,50]
kernelCategorical'linear', 'poly', 'rbf', 'sigmoid'
KNN Classifiern_neighborsDiscrete[1,20]
ANN ClassifieroptimizerCategorical'adam', 'rmsprop', 'sgd'
activationCategorical'relu', 'tanh'
batch_sizeDiscrete[16,64]
neuronsDiscrete[10,100]
epochsDiscrete[20,50]
patienceDiscrete[3,20]
RF Regressorn_estimatorsDiscrete[10,100]
max_depthDiscrete[5,50]
min_samples_splitDiscrete[2,11]
min_samples_leafDiscrete[1,11]
criterionCategorical'mse', 'mae'
max_featuresDiscrete[1,13]
SVM RegressorCContinuous[0.1,50]
kernelCategorical'linear', 'poly', 'rbf', 'sigmoid'
epsilonContinuous[0.001,1]
KNN Regressorn_neighborsDiscrete[1,20]
ANN RegressoroptimizerCategorical'adam', 'rmsprop'
activationCategorical'relu', 'tanh'
lossCategorical'mse', 'mae'
batch_sizeDiscrete[16,64]
neuronsDiscrete[10,100]
epochsDiscrete[20,50]
patienceDiscrete[3,20]

HPO Algorithms

Requirements

Contact-Info

Please feel free to contact me for any questions or cooperation opportunities. I'd be happy to help.

Citation

If you find this repository useful in your research, please cite this article as:

L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.

@article{YANG2020295,
title = "On hyperparameter optimization of machine learning algorithms: Theory and practice",
author = "Li Yang and Abdallah Shami",
volume = "415",
pages = "295 - 316",
journal = "Neurocomputing",
year = "2020",
issn = "0925-2312",
doi = "https://doi.org/10.1016/j.neucom.2020.07.061",
url = "http://www.sciencedirect.com/science/article/pii/S0925231220311693"
}