Evolutionary Optimization of Hyperparameters in Deep Learning Models

Jin Young Kim, Sung-Bae Cho

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Recently, deep learning is one of the most popular techniques in artificial intelligence. However, to construct a deep learning model, various components must be set up, including activation functions, optimization methods, a configuration of model structure called hyperparameters. As they affect the performance of deep learning, researchers are working hard to find optimal hyperparameters when solving problems with deep learning. Activation function and optimization technique play a crucial role in the forward and backward processes of model learning, but they are set up in a heuristic way. The previous studies have been conducted to optimize either activation function or optimization technique, while the relationship between them is neglected to search them at the same time. In this paper, we propose a novel method based on genetic programming to simultaneously find the optimal activation functions and optimization techniques. In genetic programming, each individual is composed of two chromosomes, one for the activation function and the other for the optimization technique. To calculate the fitness of one individual, we construct a neural network with the activation function and optimization technique that the individual represents. The deep learning model found through our method has 82.59% and 53.04% of accuracies for the CIFAR-10 and CIFAR-100 datasets, which outperforms the conventional methods. Moreover, we analyze the activation function found and confirm the usefulness of the proposed method.

Original languageEnglish
Title of host publication2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages831-837
Number of pages7
ISBN (Electronic)9781728121536
DOIs
Publication statusPublished - 2019 Jun 1
Event2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Wellington, New Zealand
Duration: 2019 Jun 102019 Jun 13

Publication series

Name2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings

Conference

Conference2019 IEEE Congress on Evolutionary Computation, CEC 2019
CountryNew Zealand
CityWellington
Period19/6/1019/6/13

Fingerprint

Hyperparameters
Evolutionary Optimization
Activation Function
Chemical activation
Optimization Techniques
Genetic programming
Genetic Programming
Model
Function Optimization
Component Model
Chromosomes
Model structures
Learning
Deep learning
Fitness
Chromosome
Artificial intelligence
Optimization Methods
Artificial Intelligence
Optimise

All Science Journal Classification (ASJC) codes

  • Computational Mathematics
  • Modelling and Simulation

Cite this

Kim, J. Y., & Cho, S-B. (2019). Evolutionary Optimization of Hyperparameters in Deep Learning Models. In 2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings (pp. 831-837). [8790354] (2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CEC.2019.8790354
Kim, Jin Young ; Cho, Sung-Bae. / Evolutionary Optimization of Hyperparameters in Deep Learning Models. 2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 831-837 (2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings).
@inproceedings{ba6a5c4f7f9c43559eba3b7a8a5fddfb,
title = "Evolutionary Optimization of Hyperparameters in Deep Learning Models",
abstract = "Recently, deep learning is one of the most popular techniques in artificial intelligence. However, to construct a deep learning model, various components must be set up, including activation functions, optimization methods, a configuration of model structure called hyperparameters. As they affect the performance of deep learning, researchers are working hard to find optimal hyperparameters when solving problems with deep learning. Activation function and optimization technique play a crucial role in the forward and backward processes of model learning, but they are set up in a heuristic way. The previous studies have been conducted to optimize either activation function or optimization technique, while the relationship between them is neglected to search them at the same time. In this paper, we propose a novel method based on genetic programming to simultaneously find the optimal activation functions and optimization techniques. In genetic programming, each individual is composed of two chromosomes, one for the activation function and the other for the optimization technique. To calculate the fitness of one individual, we construct a neural network with the activation function and optimization technique that the individual represents. The deep learning model found through our method has 82.59{\%} and 53.04{\%} of accuracies for the CIFAR-10 and CIFAR-100 datasets, which outperforms the conventional methods. Moreover, we analyze the activation function found and confirm the usefulness of the proposed method.",
author = "Kim, {Jin Young} and Sung-Bae Cho",
year = "2019",
month = "6",
day = "1",
doi = "10.1109/CEC.2019.8790354",
language = "English",
series = "2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "831--837",
booktitle = "2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings",
address = "United States",

}

Kim, JY & Cho, S-B 2019, Evolutionary Optimization of Hyperparameters in Deep Learning Models. in 2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings., 8790354, 2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings, Institute of Electrical and Electronics Engineers Inc., pp. 831-837, 2019 IEEE Congress on Evolutionary Computation, CEC 2019, Wellington, New Zealand, 19/6/10. https://doi.org/10.1109/CEC.2019.8790354

Evolutionary Optimization of Hyperparameters in Deep Learning Models. / Kim, Jin Young; Cho, Sung-Bae.

2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. p. 831-837 8790354 (2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Evolutionary Optimization of Hyperparameters in Deep Learning Models

AU - Kim, Jin Young

AU - Cho, Sung-Bae

PY - 2019/6/1

Y1 - 2019/6/1

N2 - Recently, deep learning is one of the most popular techniques in artificial intelligence. However, to construct a deep learning model, various components must be set up, including activation functions, optimization methods, a configuration of model structure called hyperparameters. As they affect the performance of deep learning, researchers are working hard to find optimal hyperparameters when solving problems with deep learning. Activation function and optimization technique play a crucial role in the forward and backward processes of model learning, but they are set up in a heuristic way. The previous studies have been conducted to optimize either activation function or optimization technique, while the relationship between them is neglected to search them at the same time. In this paper, we propose a novel method based on genetic programming to simultaneously find the optimal activation functions and optimization techniques. In genetic programming, each individual is composed of two chromosomes, one for the activation function and the other for the optimization technique. To calculate the fitness of one individual, we construct a neural network with the activation function and optimization technique that the individual represents. The deep learning model found through our method has 82.59% and 53.04% of accuracies for the CIFAR-10 and CIFAR-100 datasets, which outperforms the conventional methods. Moreover, we analyze the activation function found and confirm the usefulness of the proposed method.

AB - Recently, deep learning is one of the most popular techniques in artificial intelligence. However, to construct a deep learning model, various components must be set up, including activation functions, optimization methods, a configuration of model structure called hyperparameters. As they affect the performance of deep learning, researchers are working hard to find optimal hyperparameters when solving problems with deep learning. Activation function and optimization technique play a crucial role in the forward and backward processes of model learning, but they are set up in a heuristic way. The previous studies have been conducted to optimize either activation function or optimization technique, while the relationship between them is neglected to search them at the same time. In this paper, we propose a novel method based on genetic programming to simultaneously find the optimal activation functions and optimization techniques. In genetic programming, each individual is composed of two chromosomes, one for the activation function and the other for the optimization technique. To calculate the fitness of one individual, we construct a neural network with the activation function and optimization technique that the individual represents. The deep learning model found through our method has 82.59% and 53.04% of accuracies for the CIFAR-10 and CIFAR-100 datasets, which outperforms the conventional methods. Moreover, we analyze the activation function found and confirm the usefulness of the proposed method.

UR - http://www.scopus.com/inward/record.url?scp=85071311502&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071311502&partnerID=8YFLogxK

U2 - 10.1109/CEC.2019.8790354

DO - 10.1109/CEC.2019.8790354

M3 - Conference contribution

T3 - 2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings

SP - 831

EP - 837

BT - 2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Kim JY, Cho S-B. Evolutionary Optimization of Hyperparameters in Deep Learning Models. In 2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2019. p. 831-837. 8790354. (2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings). https://doi.org/10.1109/CEC.2019.8790354