Showing papers on "Deep learning published in 1988"

Neural net pruning-why and how

Abstract: A continuing question in neural net research is the size of network needed to solve a particular problem. If training is started with too small a network for the problem no learning can occur. The researcher must then go through a slow process of deciding that no learning is taking place, increasing the size of the network and training again. If a network that is larger than required is used, then processing is slowed, particularly on a conventional von Neumann computer. An approach to this problem is discussed that is based on learning with a net which is larger than the minimum size network required to solve the problem and then pruning the solution network. The result is a small, efficient network that performs as well or better than the original which does not give a complete answer to the question, since the size of the initial network is still largely based on guesswork but it gives a very useful partial answer and sheds some light on the workings of a neural network in the process. >

Efficient Parallel Learning Algorithms for Neural Networks

Abstract: Parallelizable optimization techniques are applied to the problem of learning in feedforward neural networks. In addition to having superior convergence properties, optimization techniques such as the Polak-Ribiere method are also significantly more efficient than the Backpropagation algorithm. These results are based on experiments performed on small boolean learning problems and the noisy real-valued learning problem of hand-written character recognition.

Neural networks. II. What are they and why is everybody so interested in them now

Abstract: For part I, see ibid., vol. EX-2, no. 4, p. 10-11 (1987). The learning ability of neural networks and their ability to generalize and to abstract or generate ideals from an imperfect training set are examined. Their potential for multiprocessing is considered. A brief history of neural network research is followed by a discussion of their architectures and a presentation of several specific architectures and learning techniques. The Cauchy machine, which represents a possible solution to the local minima problem encountered with virtually every other neural network training algorithm, is described. The outlook for neural nets is briefly considered. >

Learning state space trajectories in recurrent neural networks : a preliminary report.

Abstract: : We describe a procedure for finding learning state space trajectories in recurrent neural networks. Keywords: Connectionism; Learning algorithm; Trajectories following; Minimizing functionals.

Algorithm Development For Neural Networks

Abstract: This paper is concerned with the development of algorithms for solving optimiza­ tion problems with a network of artificial neurons. Although there is no notion of step-by-step sequencing in a neural network, it is possible to develop tools and techniques for interconnecting a network of neurons so that it will achieve stable states corresponding to possible solutions of a problem. An approach to implementing an important class of constraints in a network of artificial neurons is presented and illustrated by developing a solution to a resource allocation problem. 1. A neural network is a collection of highly-interconnected, simple analog process­ ing elements that mimic biological neurons. Although neural networks are greatly sim­ plified models of biological neural systems, they possess computational properties that are applicable to a variety of problems including speech and image processing, pattern recognition, cognitive information processing and combinatorial optimization.Neural computing relies upon massive parallelism and a high degree of connectivity among simple analog processors and therefore represents a radical departure from the von Neumann architecture. Unlike von Neumann machines, neural computations have no notion of step-by-step sequencing. Instead, computations result from the collective emergent behavior of a dynamical system of analog elements. While neural elements can assume continuous values between 0 and 1 as the computation progresses, they are usually designed to settle eventually into binary-valued states which collectively correspond to problem solutions. Neural networks, therefore, bear little resemblance to the analog processing that was popular in the 1950's.Much current research in neural networks is directed toward exploring their poten­ tial to produce intelligent behavior. The use of neural networks to produce intelli­ gent behavior has already been demonstrated in a variety of areas such as feature extraction and pattern classification. An equally important application of neural net­ works, pioneered by Hopfield and Tank,l>2 is the solution of optimization problems by simultaneously evaluating multiple competing hypotheses. Neural networks can rapidly produce good, though not necessarily optimal, solutions to such problems3.4.

Neural networks: translation-, rotation- and scale-invariant pattern recognition

Abstract: A neural network model which is capable of recognising transformed versions of a set of learnt patterns is proposed The group of transformations includes global translations, rotations and scale transformations The neural firing thresholds are used as additional degrees of freedom

Mapping Classifier Systems Into Neural Networks

Abstract: Classifier systems are machine learning systems incotporating a genetic algorithm as the learning mechanism. Although they respond to inputs that neural networks can respond to, their internal structure, representation formalisms, and learning mechanisms differ markedly from those employed by neural network researchers in the same sorts of domains. As a result, one might conclude that these two types of machine learning formalisms are intrinsically different. This is one of two papers that, taken together, prove instead that classifier systems and neural networks are equivalent. In this paper, half of the equivalence is demonstrated through the description of a transformation procedure that will map classifier systems into neural networks that are isomorphic in behavior. Several alterations on the commonly-used paradigms employed by neural network researchers are required in order to make the transformation work. These alterations are noted and their appropriateness is discussed. The paper concludes with a discussion of the practical import of these results, and with comments on their extensibility.

Neural networks in defense applications

Abstract: The author presents early results of preliminary experimental investigations of the performance of various trainable (back-propagation) networks applied to sensor signal processing and optimization processing problems. Various network topologies and target signatures were exercised. Networks ranged from two-layer to six-layer, with varying number of neurons per layer. Multiple training and test sets were synthesized and used in evaluating both the training characteristics and processing performance of the various networks. Preliminary results for learning networks in pattern recognition applications indicate very promising performance characteristics for fairly simple back-propagation networks, on the order of less than 50 neurons, as a function of topology, learning rate, and sensor signal complexity. Overall, the networks behaved as expected for back-propagation networks. >

Neural network learning algorithms

Abstract: The earliest network models of associative memory were based on correlations between input and output patterns of activity in linear processing units. These models have several features that make them attractive: The synaptic strengths are computed from information available locally at each synapse in a single trial; the information is distributed in a large number of connection strengths, the recall of stored information is associative, and the network can generalize to new input patterns that are similar to stored patterns. There are also severe limitations with this class of linear associative matrix models, including interference between stored items, especially between ones that are related, and inability to make decisions that are contingent on several inputs. New neural network models and neural network learning algorithms have been introduced recently that overcome some of the shortcomings of the associative matrix models of memory. These learning algorithms require many training examples to create the internal representations needed to perform a difficult task and generalize properly. They share some properties with human skill acquisition.

Neural network computation

Abstract: Since 1982 there has been an enormous resurgence of interest in the possibility of making trainable, general purpose pattern recognition machines which are intended to mimic some of the processing abilities of human brains. A neural network 'computer' consists of a set of processing units (or artificial neurons) joined together by a set of weighted connections. Such networks are programmed by applying training patterns which fix the output states of some or all of the units, and a learning algorithm then adjusts the connections in response to the training patterns. The author provides an introduction to the field and to the current state of the art.

Applications and analysis of second order artificial neural networks

Abstract: The authors investigate a two-layer artificial neural network which consists of one hidden layer of second-order neurons and an output layer of first-order neurons. The second-order neurons yield conic-surface types of decision regions such as ellipses and parabolas. A simulation example is presented. It is shown that such networks yield more flexibility than first-order networks. An analysis of the memory capacities of single-layer and two-layer networks is presented. >

A Self-Learning Neural Network

Abstract: We propose a new neural network structure that is compatible with silicon technology and has built-in learning capability. The thrust of this network work is a new synapse function. The synapses have the feature that the learning parameter is embodied in the thresholds of MOSFET devices and is local in character. The network is shown to be capable of learning by example as well as exhibiting the desirable features of the Hopfield type networks.

Analog neural systems

Abstract: Summary form only given. The author describes work in understanding the limitations of analog computing systems and their relevance to several problems involving neural networks. Two main themes are considered, both involving computational complexity considerations. The first theme involves the proposal that neural networks be used to solve the 'intractable' (NP-complete) combinatorial optimization problem. The second theme involves adaptation or learning algorithms for neural networks. >

Neural models and learning rules

Abstract: The basic computing element in models of neural networks that focus on information processing capabilities is a 'neural' unit that has an output that is a function of the sum of its inputs. Information is stored in 'synapses' or connection strengths between units. Networks of these neurons are not programmed like standard computers, but trained by data input. The author examines both unsupervised learning algorithms, which allow networks to find correlations in the input, and supervised learning algorithms, which allow the pairing of arbitrary patterns. >