Committee machine

About: Committee machine is a research topic. Over the lifetime, 272 publications have been published within this topic receiving 4554 citations.

A Bayesian Committee Machine

Abstract: The Bayesian committee machine (BCM) is a novel approach to combining estimators that were trained on different data sets. Although the BCM can be applied to the combination of any kind of estimators, the main foci are gaussian process regression and related systems such as regularization networks and smoothing splines for which the degrees of freedom increase with the number of training data. Somewhat surprisingly, we find that the performance of the BCM improves if several test points are queried at the same time and is optimal if the number of test points is at least as large as the degrees of freedom of the estimator. The BCM also provides a new solution for on-line learning with potential applications to data mining. We apply the BCM to systems with fixed basis functions and discuss its relationship to gaussian process regression. Finally, we show how the ideas behind the BCM can be applied in a non-Bayesian setting to extend the input-dependent combination of estimators.

On-line learning in soft committee machines

Abstract: The problem of on-line learning in two-layer neural networks is studied within the framework of statistical mechanics. A fully connected committee machine with K hidden units is trained by gradient descent to perform a task defined by a teacher committee machine with M hidden units acting on randomly drawn inputs. The approach, based on a direct averaging over the activation of the hidden units, results in a set of first-order differential equations that describes the dynamical evolution of the overlaps among the various hidden units and allows for a computation of the generalization error. The equations of motion are obtained analytically for general K and M and provide a powerful tool used here to study a variety of realizable, over-realizable, and unrealizable learning scenarios and to analyze the role of the learning rate in controlling the evolution and convergence of the learning process.

Committee neural networks for porosity and permeability prediction from well logs

Abstract: Neural computing has moved beyond simple demonstration to more significant applications. Encouraged by recent developments in artificial neural network (ANN) modelling techniques, we have developed committee machine (CM) networks for converting well logs to porosity and permeability, and have applied the networks to real well data from the North Sea. Simple three-layer back-propagation ANNs constitute the blocks of a modular system where the porosity ANN uses sonic, density and resistivity logs for input. The permeability ANN is slightly more complex, with four inputs (density, gamma ray, neutron porosity and sonic). The optimum size of the hidden layer, the number of training data required, and alternative training techniques have been investigated using synthetic logs. For both networks an optimal number of neurons in the hidden layer is in the range 8–10. With a lower number of hidden units the network fails to represent the problem, and for higher complexity overfitting becomes a problem when data are noisy. A sufficient number of training samples for the porosity ANN is around 150, while the permeability ANN requires twice as many in order to keep network errors well below the errors in core data. For the porosity ANN the overtraining strategy is the suitable technique for bias reduction and an unconstrained optimal linear combination (OLC) is the best method of combining the CM output. For permeability, on the other hand, the combination of overtraining and OLC does not work. Error reduction by validation, simple averaging combined with range-splitting provides the required accuracy. The accuracy of the resulting CM is restricted only by the accuracy of the real data. The ANN approach is shown to be superior to multiple linear regression techniques even with minor non-linearity in the background model.

Learning by on-line gradient descent

Abstract: We study on-line gradient-descent learning in multilayer networks analytically and numerically. The training is based on randomly drawn inputs and their corresponding outputs as defined by a target rule. In the thermodynamic limit we derive deterministic differential equations for the order parameters of the problem which allow an exact calculation of the evolution of the generalization error. First we consider a single-layer perceptron with sigmoidal activation function learning a target rule defined by a network of the same architecture. For this model the generalization error decays exponentially with the number of training examples if the learning rate is sufficiently small. However, if the learning rate is increased above a critical value, perfect learning is no longer possible. For architectures with hidden layers and fixed hidden-to-output weights, such as the parity and the committee machine, we find additional effects related to the existence of symmetries in these problems.