Awesome
<div align="center"><img src="https://raw.githubusercontent.com/optuna/optuna/master/docs/image/optuna-logo.png" width="800"/></div>Optuna: A hyperparameter optimization framework
:link: Website | :page_with_curl: Docs | :gear: Install Guide | :pencil: Tutorial | :bulb: Examples | Twitter | LinkedIn | Medium
Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, define-by-run style user API. Thanks to our define-by-run API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters.
:loudspeaker: News
<!-- TODO: when you add a new line, please delete the oldest line -->- Sep 2, 2024: Optuna 4.0 is available! You can install it by
pip install -U optuna
. Find the latest here and check our article. - Aug 30, 2024: We posted an article about the official release of OptunaHub.
- Aug 28, 2024: We posted an article about implicit natural gradient optimization (
INGO
), a sampler newly supported in OptunaHub. - Aug 20, 2024: We posted an article about
ArtifactStore
officially supported from Optuna 4.0. - Aug 15, 2024: We posted an article about
CatCMA
, a sampler newly supported in OptunaHub. - Aug 7, 2024: We posted an article about the significant (300x) speed up of multi-objective
TPESampler
in Optuna 4.0.
:fire: Key Features
Optuna has modern functionalities as follows:
- Lightweight, versatile, and platform agnostic architecture
- Handle a wide variety of tasks with a simple installation that has few requirements.
- Pythonic search spaces
- Define search spaces using familiar Python syntax including conditionals and loops.
- Efficient optimization algorithms
- Adopt state-of-the-art algorithms for sampling hyperparameters and efficiently pruning unpromising trials.
- Easy parallelization
- Scale studies to tens or hundreds of workers with little or no changes to the code.
- Quick visualization
- Inspect optimization histories from a variety of plotting functions.
Basic Concepts
We use the terms study and trial as follows:
- Study: optimization based on an objective function
- Trial: a single execution of the objective function
Please refer to the sample code below. The goal of a study is to find out the optimal set of
hyperparameter values (e.g., regressor
and svr_c
) through multiple trials (e.g.,
n_trials=100
). Optuna is a framework designed for automation and acceleration of
optimization studies.
import ...
# Define an objective function to be minimized.
def objective(trial):
# Invoke suggest methods of a Trial object to generate hyperparameters.
regressor_name = trial.suggest_categorical('regressor', ['SVR', 'RandomForest'])
if regressor_name == 'SVR':
svr_c = trial.suggest_float('svr_c', 1e-10, 1e10, log=True)
regressor_obj = sklearn.svm.SVR(C=svr_c)
else:
rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32)
regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth)
X, y = sklearn.datasets.fetch_california_housing(return_X_y=True)
X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0)
regressor_obj.fit(X_train, y_train)
y_pred = regressor_obj.predict(X_val)
error = sklearn.metrics.mean_squared_error(y_val, y_pred)
return error # An objective value linked with the Trial object.
study = optuna.create_study() # Create a new study.
study.optimize(objective, n_trials=100) # Invoke optimization of the objective function.
</details>
[!NOTE] More examples can be found in optuna/optuna-examples.
The examples cover diverse problem setups such as multi-objective optimization, constrained optimization, pruning, and distributed optimization.
Installation
Optuna is available at the Python Package Index and on Anaconda Cloud.
# PyPI
$ pip install optuna
# Anaconda Cloud
$ conda install -c conda-forge optuna
[!IMPORTANT] Optuna supports Python 3.7 or newer.
Also, we provide Optuna docker images on DockerHub.
Integrations
Optuna has integration features with various third-party libraries. Integrations can be found in optuna/optuna-integration and the document is available here.
<details> <summary>Supported integration libraries</summary>- Catboost
- Dask
- fastai
- Keras
- LightGBM
- MLflow
- PyTorch
- PyTorch Ignite
- PyTorch Lightning
- TensorBoard
- TensorFlow
- tf.keras
- Weights & Biases
- XGBoost
Web Dashboard
Optuna Dashboard is a real-time web dashboard for Optuna. You can check the optimization history, hyperparameter importance, etc. in graphs and tables. You don't need to create a Python script to call Optuna's visualization functions. Feature requests and bug reports are welcome!
optuna-dashboard
can be installed via pip:
$ pip install optuna-dashboard
<details> <summary>Sample code to launch Optuna Dashboard</summary>[!TIP] Please check out the convenience of Optuna Dashboard using the sample code below.
Save the following code as optimize_toy.py
.
import optuna
def objective(trial):
x1 = trial.suggest_float("x1", -100, 100)
x2 = trial.suggest_float("x2", -100, 100)
return x1 ** 2 + 0.01 * x2 ** 2
study = optuna.create_study(storage="sqlite:///db.sqlite3") # Create a new study with database.
study.optimize(objective, n_trials=100)
Then try the commands below:
# Run the study specified above
$ python optimize_toy.py
# Launch the dashboard based on the storage `sqlite:///db.sqlite3`
$ optuna-dashboard sqlite:///db.sqlite3
...
Listening on http://localhost:8080/
Hit Ctrl-C to quit.
</details>
OptunaHub
OptunaHub is a feature-sharing platform for Optuna. You can use the registered features and publish your packages.
Use registered features
optunahub
can be installed via pip:
$ pip install optunahub
You can load registered module with optunahub.load_module
.
import optuna
import optunahub
def objective(trial: optuna.Trial) -> float:
x = trial.suggest_float("x", 0, 1)
return x
mod = optunahub.load_module("samplers/simulated_annealing")
study = optuna.create_study(sampler=mod.SimulatedAnnealingSampler())
study.optimize(objective, n_trials=20)
print(study.best_trial.value, study.best_trial.params)
For more details, please refer to the optunahub documentation.
Publish your packages
You can publish your package via optunahub-registry. See the OptunaHub tutorial.
Communication
- GitHub Discussions for questions.
- GitHub Issues for bug reports and feature requests.
Contribution
Any contributions to Optuna are more than welcome!
If you are new to Optuna, please check the good first issues. They are relatively simple, well-defined, and often good starting points for you to get familiar with the contribution workflow and other developers.
If you already have contributed to Optuna, we recommend the other contribution-welcome issues.
For general guidelines on how to contribute to the project, take a look at CONTRIBUTING.md.
Reference
If you use Optuna in one of your research projects, please cite our KDD paper "Optuna: A Next-generation Hyperparameter Optimization Framework":
<details open> <summary>BibTeX</summary>@inproceedings{akiba2019optuna,
title={{O}ptuna: A Next-Generation Hyperparameter Optimization Framework},
author={Akiba, Takuya and Sano, Shotaro and Yanase, Toshihiko and Ohta, Takeru and Koyama, Masanori},
booktitle={The 25th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining},
pages={2623--2631},
year={2019}
}
</details>
License
MIT License (see LICENSE).
Optuna uses the codes from SciPy and fdlibm projects (see LICENSE_THIRD_PARTY).