An Analysis of Neural Architecture Search and Hyper Parameter Optimization Methods

Hyperparameter Optimization for Convolutional Neural Networks with Genetic Algorithms and Bayesian Optimization

Deep learning (DL) has emerged as a leading approach to solving complex problems in multiple areas. Its success is mainly due to technological development, large volumes of data, and various algorithms based on artificial neural networks (ANN). However, their functioning and performance are sensitive to the configuration of hyperparameters, which result from trial and error processes based on the knowledge and creativity of experts. In this sense, to reduce subjective associated with human intervention, an approach suggests tackling the tuning of hyper-parameters as an optimization problem. Here, the objective is to find a configuration that maximizes some performance metrics. The present work presents an implementation and comparison of three optimization strategies (Random Search, Bayesian Opti-mization, and Genetic Algorithm) for image classification in the CIFAR-10 dataset using convolutional neural networks, showing that algorithm performance and their solutions are tightly bound with the kind of architecture and search space.

Deep neural networks (DNNs) show great performance in lots of applications. Convolutional neural network (CNN) is one of the classic DNNs, and various modified CNNs have been brought up, such as DenseNet, GoogleNet, ResNet, etc. For diverse tasks, a unique structure of CNN may show its advantage. However, how to design an effective CNN model for a practical task is a puzzle. In this paper, we model the architecture optimization of CNN as an optimization problem and design a Genetic network programming based Fast evolutionary learning (GNP-FEL) to optimize CNN. GNP-FEL contains three main ideas: First GNP is adopted to optimize CNN architecture and hyperparameters, which can build diverse network structures and make network parameters selfevolve; Second multi-objective optimization is designed by balancing both CNN model efficiency and structure compactness; Last a novel incremental training method is proposed to train offspring CNN models in GNP, which is capable of reducing time complexity sharply. Experiments have validated that GNP-FEL can quickly evolve a CNN classifier with a sufficiently compact architecture. And the classifier has a comparable classification effect to state-ofthe-art CNN model.

https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/cje.2020.09.007

Yi Liu

Papers

A Fast Evolutionary Learning to Optimize CNN