Incremental neural network construction algorithms for training multilayer perceptrons

Abstract

The problem of determining the architecture of a multilayer perceptron together with the disadvantages of the standard backpropagation algorithm, directed the research towards algorithms that determine not only the weights but also the structure of the net~vork necessary for learning the data. In this work we propose two algorithms: the Constructive Algorithm using Statistical Tests (CAST), and Constructive Algorithm with Multiple Operators using Statistical Tests (MOST). The first one constructs a single hidden layer network by adding hidden nodes one by one. The algorithm checks the difference between the errors of the current and candidate networks and decides whether to select the candidate network or not by using a statistical test for comparing the accuracies of the two networks. The networks that are constructed by MOST can have more than one hidden layer. The algorithm uses node removal, addition and layer addition and determines the number of nodes in layers by heuristics. To our krowledge, MOST is the only algorithm that constructs a multilayer perceptron with multiple hidden layers with multiple units per layer. The results of the algorithms are promising and near optimal.