The Study of the Hyperparameter Optimization Analysis in the Convolutional Neural Networks Model with Starfish Optimization Algorithm

Abstract

In this study, the hyperparameters of Convolutional Neural Networks (CNNs) have been optimized with the newly proposed Starfish Optimization Algorithm (SFOA) in recent years. CNN has complex hyperparameters due to its structure. In the literature, the values of hyperparameters are mostly tried to be determined with combinatorial methods. The success of metaheuristic algorithms in optimizing the variables of different problems has inspired this study. Thus, four different numbers of channel values (8, 16, 32, and 64), five different kernel size values (1x1, 3x3, 5x5, 7x7, and 9x9), four different batch size values (32, 64, 128, and 256), twenty different values randomly generated between 0 and 0.05 for the learning rate, three different optimizer types (sgdm, adam, and rmsprop), and four different epoch values (5, 10, 15, and 20), which are the most critical hyperparameters in CNN, have been determined. A 6dimensional solution space was determined with SFOA, and these hyperparameter values were placed in discretely defined dimensions. SFOA tried to determine the most appropriate hyperparameter values for the CNN model in each iteration. In this study, two different image datasets (MNIST and Kuzushiji-MNIST) were selected for CNN classification. Due to the hyperparameter optimization carried out with the SFOA algorithm, an accuracy of 99.52% for the MNIST dataset and 97.91% for the Kuzushiji-MNIST dataset was achieved. Comparisons with existing literature demonstrate that the proposed model showcases successful and competitive performance. Finally, the proposed CNN models are evaluated on a different image dataset, EMNIST (Extended MNIST). EMNIST is a more comprehensive version of MNIST developed for classifying handwritten letters and numbers. The accuracy results on the EMNIST dataset were 88.65% (the proposed CNN model with similar hyperparameter settings as MNIST) and 88.73% (the proposed CNN model with similar hyperparameter settings as Kuzushiji-MNIST), respectively. Additionally, hyperparameters for the EMNIST dataset were determined using SFOA, achieving an accuracy of 88.71%. Analyzing the hyperparameters of three different CNN models, it was observed that similar optimizer types, epoch numbers, kernel sizes, and channel numbers were preferred. This demonstrates that SFOA can produce reliable and effective settings across different datasets.