Evolutionary Optimization of Hyperparameters in Deep Learning Models

Jin Young Kim, Sung Bae Cho

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Recently, deep learning is one of the most popular techniques in artificial intelligence. However, to construct a deep learning model, various components must be set up, including activation functions, optimization methods, a configuration of model structure called hyperparameters. As they affect the performance of deep learning, researchers are working hard to find optimal hyperparameters when solving problems with deep learning. Activation function and optimization technique play a crucial role in the forward and backward processes of model learning, but they are set up in a heuristic way. The previous studies have been conducted to optimize either activation function or optimization technique, while the relationship between them is neglected to search them at the same time. In this paper, we propose a novel method based on genetic programming to simultaneously find the optimal activation functions and optimization techniques. In genetic programming, each individual is composed of two chromosomes, one for the activation function and the other for the optimization technique. To calculate the fitness of one individual, we construct a neural network with the activation function and optimization technique that the individual represents. The deep learning model found through our method has 82.59% and 53.04% of accuracies for the CIFAR-10 and CIFAR-100 datasets, which outperforms the conventional methods. Moreover, we analyze the activation function found and confirm the usefulness of the proposed method.

Original languageEnglish
Title of host publication2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages831-837
Number of pages7
ISBN (Electronic)9781728121536
DOIs
Publication statusPublished - 2019 Jun
Event2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Wellington, New Zealand
Duration: 2019 Jun 102019 Jun 13

Publication series

Name2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings

Conference

Conference2019 IEEE Congress on Evolutionary Computation, CEC 2019
CountryNew Zealand
CityWellington
Period19/6/1019/6/13

All Science Journal Classification (ASJC) codes

  • Computational Mathematics
  • Modelling and Simulation

Fingerprint Dive into the research topics of 'Evolutionary Optimization of Hyperparameters in Deep Learning Models'. Together they form a unique fingerprint.

  • Cite this

    Kim, J. Y., & Cho, S. B. (2019). Evolutionary Optimization of Hyperparameters in Deep Learning Models. In 2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings (pp. 831-837). [8790354] (2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CEC.2019.8790354