A Study on Generalization of Random Weight Network With Flat Loss

Abstract

In the scheme of learning which adjusts model parameters by minimizing a loss function, there is a conjecture that the loss function with flatter minimum may correlate with better stability and generalization of the model. This paper provides experimental evidence within the Random Weight Network (RWN)/Extreme Learning Machine (ELM) framework and further develops a theoretical analysis linking flatness to the local generalization error upper bound by deriving the RWN loss as a quadratic polynomial with respect to random weights and representing the flatness as the maximum eigenvalue of a semi-positive definite matrix. By adjusting the random weights using a genetic algorithm, where the fitness function is defined as the flatness, we validate on 10 benchmark datasets within the ELM framework that flatter loss indeed improves the model's generalization ability. The improvement size depends on the specific characteristics of datasets, particularly, on the relative decrease of maximum eigenvalues. This study shows that RWN generalization performance can be improved by optimizing random weight selection.