Publisher Summary This chapter provides an account of different neural network architectures for pattern recognition. A neural network consists of several simple processing elements called neurons. Each neuron is connected to some other neurons and possibly to the input nodes. Neural networks provide a simple computing paradigm to perform complex recognition tasks in real time. The chapter categorizes neural networks into three types: single-layer networks, multilayer feedforward networks, and feedback networks. It discusses the gradient descent and the relaxation method as the two underlying mathematical themes for deriving learning algorithms. A lot of research activity is centered on learning algorithms because of their fundamental importance in neural networks. The chapter discusses two important directions of research to improve learning algorithms: the dynamic node generation, which is used by the cascade correlation algorithm; and designing learning algorithms where the choice of parameters is not an issue. It closes with the discussion of performance and implementation issues.

Neural Networks for Pattern Recognition

From the Publisher:
An Introduction to Nueral Networks will be warmly welcomed by a wide readership seeking an authoritative treatment of this key subject without an intimidating level of mathematics in the presentation.

An introduction to neural networks

From the Publisher:
As book review editor of the IEEE Transactions on Neural Networks, Mohamad Hassoun has had the opportunity to assess the multitude of books on artificial neural networks that have appeared in recent years. Now, in Fundamentals of Artificial Neural Networks, he provides the first systematic account of artificial neural network paradigms by identifying clearly the fundamental concepts and major methodologies underlying most of the current theory and practice employed by neural network researchers.
Such a systematic and unified treatment, although sadly lacking in most recent texts on neural networks, makes the subject more accessible to students and practitioners. Here, important results are integrated in order to more fully explain a wide range of existing empirical observations and commonly used heuristics. There are numerous illustrative examples, over 200 end-of-chapter analytical and computer-based problems that will aid in the development of neural network analysis and design skills, and a bibliography of nearly 700 references.
Proceeding in a clear and logical fashion, the first two chapters present the basic building blocks and concepts of artificial neural networks and analyze the computational capabilities of the basic network architectures involved. Supervised, reinforcement, and unsupervised learning rules in simple nets are brought together in a common framework in chapter three. The convergence and solution properties of these learning rules are then treated mathematically in chapter four, using the "average learning equation" analysis approach. This organization of material makes it natural to switch into learning multilayer nets using backpropand its variants, described in chapter five. Chapter six covers most of the major neural network paradigms, while associative memories and energy minimizing nets are given detailed coverage in the next chapter. The final chapter takes up Boltzmann machines and Boltzmann learning along with other global search/optimization algorithms such as stochastic gradient search, simulated annealing, and genetic algorithms.

/pdf/fundamentals-of-artificial-neural-networks-2ae4w3uadi.pdf

Fundamentals of artificial neural networks

Stability and encoding properties of two-layer nonlinear feedback neural networks are examined. Bidirectionality is introduced in neural nets to produce two-way associative search for stored associations. The bidirectional associative memory (BAM) is the minimal two-layer nonlinear feedback network. The author proves that every n-by-p matrix M is a bidirectionally stable heteroassociative content-addressable memory for both binary/bipolar and continuous neurons. When the BAM neutrons are activated, the network quickly evolves to a stable state of two-pattern reverberation, or resonance. The stable reverberation corresponds to a system energy local minimum. Heteroassociative information is encoded in a BAM by summing correlation matrices. The BAM storage capacity for reliable recall is roughly m<min (n,p). It is also shown that it is better on average to use bipolar (-1,1) coding than binary

Bidirectional associative memories

Nonlinear neural networks: Principles, mechanisms, and architectures

Techniques from coding theory are applied to study rigorously the capacity of the Hopfield associative memory. Such a memory stores n -tuple of \pm 1 's. The components change depending on a hard-limited version of linear functions of all other components. With symmetric connections between components, a stable state is ultimately reached. By building up the connection matrix as a sum-of-outer products of m fundamental memories, one hopes to be able to recover a certain one of the m memories by using an initial n -tuple probe vector less than a Hamming distance n/2 away from the fundamental memory. If m fundamental memories are chosen at random, the maximum asympotic value of m in order that most of the m original memories are exactly recoverable is n/(2 \log n) . With the added restriction that every one of the m fundamental memories be recoverable exactly, m can be no more than n/(4 \log n) asymptotically as n approaches infinity. Extensions are also considered, in particular to capacity under quantization of the outer-product connection matrix. This quantized memory capacity problem is closely related to the capacity of the quantized Gaussian channel.

/pdf/the-capacity-of-the-hopfield-associative-memory-4fg6q0raa8.pdf

The capacity of the Hopfield associative memory

© Annales de l’institut Fourier, 1974, tous droits réservés. L’accès aux archives de la revue « Annales de l’institut Fourier » (http://annalif.ujf-grenoble.fr/) implique l’accord avec les conditions générales d’utilisation (http://www.numdam.org/legal.php). Toute utilisation commerciale ou impression systématique est constitutive d’une infraction pénale. Toute copie ou impression de ce fichier doit contenir la présente mention de copyright.

/pdf/monotonicity-of-certain-functionals-under-rearrangement-4kzwxcmoqi.pdf

Monotonicity of certain functionals under rearrangement

Previous work has shown that the capacity of the single-mode photon-counting channel in the region in which quantum effects are most pronounced is 1/(kT \ln 2) bit/J. Here T is the quantum noise temperature, and k is Boltzmann's constant. However, to achieve this, the number of photons per second must be very low, so this does not determine the capacity of the average power-limited photon channel in bits per second. Here the previous work is extended and the capacity is determined in bits per second. This includes a determination of the optimum counting time. The capacity in bits per second is for low T asymptotic to P_{s}/(kT \ln 2) , where P_{s} is the average signal power. This is high enough to suggest that photon counting be seriously considered for applications such as a deep space to near Earth link. Some related problems are also considered including a peak power limitation.

The capacity of the photon counting channel

Epsilon-delta entropy for probabilistic metric spaces applied to stochastic processes in data transmission

/pdf/epsilon-entropy-of-stochastic-processes-1l4pqy7hl0.pdf

Epsilon entropy of stochastic processes.

This article studies efficient data transmission, or "data compression", from the standpoint of the theory of epsilon entropy. The notion of the entropy of a "data source" is defined. This quantity gives a precise measure of the amount of channel capacity necessary to describe a data source to within a given fidelity, epsilon, with probability one, when each separate "experiment" must be transmitted without storage from experiment to experiment. We also define the absolute epsilon entropy of a source, which is the amount of capacity needed when storage of experiments is allowed before transmission. The absolute epsilon entropy is shown to be equal to Shannon's rate distortion function evaluated for zero distortion, when suitable identifications are made. The main result is that the absolute epsilon entropy and the epsilon entropy have ratio close to one if either is large. Thus, very little can be saved by storing the results of independent experiments before transmission.

/pdf/epsilon-entropy-and-data-compression-4ofr2eynj8.pdf

Eugene R. Rodemich

Papers

The capacity of the Hopfield associative memory

Monotonicity of certain functionals under rearrangement

The capacity of the photon counting channel

Epsilon entropy of stochastic processes.

Epsilon Entropy and Data Compression