This code provides a hyper-parameter optimization implementation for machine learning algorithms, as described in the paper "On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice".
To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization.
This paper and code will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
Section 3: Common hyper-parameters of machine learning algorithms
Section 4: Hyper-parameter optimization techniques introduction
Section 5: How to choose optimization techniques for different machine learning models
Section 6: Common Python libraries/tools
Section 7: Experimental results (sample code in "HPO_Regression.ipynb" and "HPO_Classification.ipynb")
Section 8: Open challenges and future research directions
Summary table for Sections 3-6: Table 2: A comprehensive overview of common ML models, their hyper-parameters, suit-able optimization techniques, and available Python libraries
Summary table for Sections 8: Table 10: The open challenges and future directions of HPO research
Sample code for hyper-parameter optimization implementation for machine learning algorithms is provided in this repository.
HPO_Regression.ipynb
Dataset used: Boston-Housing
HPO_Classification.ipynb
Dataset used: MNIST
- Random forest (RF)
- Support vector machine (SVM)
- K-nearest neighbor (KNN)
- Grid search
- Random search
- Hyperband
- Bayesian Optimization with Gaussian Processes (BO-GP)
- Bayesian Optimization with Tree-structured Parzen Estimator (BO-TPE)
- Particle swarm optimization (PSO)
- Genetic algorithm (GA).
- Python 3.5
- scikit-learn
- hyperband
- scikit-optimize
- hyperopt
- optunity
- DEAP
- TPOT
If you find this repository useful in your research, please this article as:
L. Yang, A. Shami, On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice, Neurocomputing (2020), doi: https://doi.org/10.1016/j.neucom.2020.07.061