Showing papers on "Deep learning published in 1995"

Gradient calculations for dynamic recurrent neural networks: a survey

Abstract: Surveys learning algorithms for recurrent neural networks with hidden units and puts the various techniques into a common framework. The authors discuss fixed point learning algorithms, namely recurrent backpropagation and deterministic Boltzmann machines, and nonfixed point algorithms, namely backpropagation through time, Elman's history cutoff, and Jordan's output feedback architecture. Forward propagation, an on-line technique that uses adjoint equations, and variations thereof, are also discussed. In many cases, the unified presentation leads to generalizations of various sorts. The author discusses advantages and disadvantages of temporally continuous neural networks in contrast to clocked ones continues with some "tricks of the trade" for training, using, and simulating continuous time and recurrent neural networks. The author presents some simulations, and at the end, addresses issues of computational complexity and learning speed. >

Neural Network Fundamentals with Graphs, Algorithms and Applications

Abstract: Part 1 Fundamentals: basics of neuroscience and artificial neuron models graphs algorithms. Part 2 Feedforward networks: perceptrons and LMS algorithm complexity of learning using feedforward networks adaptive structure networks. Part 3 Recurrent networks: symmetric and asymmetric recurrent network competitive learning and self-organizing networks. Part 4 Applications of neural networks: neural networks approach to solving hard problems.

Backpropagation: theory, architectures, and applications

Abstract: Contents: D.E. Rumelhart, R. Durbin, R. Golden, Y. Chauvin, Backpropagation: The Basic Theory. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, K.J. Lang, Phoneme Recognition Using Time-Delay Neural Networks. C. Schley, Y. Chauvin, V. Henkle, Automated Aircraft Flare and Touchdown Control Using Neural Networks. F.J. Pineda, Recurrent Backpropagation Networks. M.C. Mozer, A Focused Backpropagation Algorithm for Temporal Pattern Recognition. D.H. Nguyen, B. Widrow, Nonlinear Control with Neural Networks. M.I. Jordan, D.E. Rumelhart, Forward Models: Supervised Learning with a Distal Teacher. S.J. Hanson, Backpropagation: Some Comments and Variations. A. Cleeremans, D. Servan-Schreiber, J.L. McClelland, Graded State Machines: The Representation of Temporal Contingencies in Feedback Networks. S. Becker, G.E. Hinton, Spatial Coherence as an Internal Teacher for a Neural Network. J.R. Bachrach, M.C. Mozer, Connectionist Modeling and Control of Finite State Systems Given Partial State Information. P. Baldi, Y. Chauvin, K. Hornik, Backpropagation and Unsupervised Learning in Linear Networks. R.J. Williams, D. Zipser, Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity. P. Baldi, Y. Chauvin, When Neural Networks Play Sherlock Homes. P. Baldi, Gradient Descent Learning Algorithms: A Unified Perspective.

Advanced algorithms for neural networks: a C++ sourcebook

Abstract: Deterministic Optimization Stochastic Optimization Hybrid Training Algorithms Probabilistic Neural Networks I: Introduction Probabilistic Neural Networks II: Advanced Techniques Generalized Regression The Gram-Charlier Neural Network Dimension Reduction and Orthogonalization Assessing Generalization Ability Using the PNN Program.

Learning in linear neural networks: a survey

Abstract: Networks of linear units are the simplest kind of networks, where the basic questions related to learning, generalization, and self-organization can sometimes be answered analytically. We survey most of the known results on linear networks, including: 1) backpropagation learning and the structure of the error function landscape, 2) the temporal evolution of generalization, and 3) unsupervised learning algorithms and their properties. The connections to classical statistical ideas, such as principal component analysis (PCA), are emphasized as well as several simple but challenging open questions. A few new results are also spread across the paper, including an analysis of the effect of noise on backpropagation networks and a unified view of all unsupervised algorithms. >

Neural Networks for Chemical Engineers

Abstract: Part 1 Theoretical issues: an introduction to artificial neural networks, D.T. Pham unsupervised neural learning, E. Oja back-propagation and its variations, D. Tsaptsinos the general approximation problem for feed-forward neural networks, A.B. Bulsari connectionism vs symbolism - an overview, A.B. Bujosa et al introduction to connectionist computer vision systems, D.W. Moolman et al. Part 2 Applications: lime kiln simulation and control by neural networks, B. Ribeiro and A. Dourado Correia concentration estimation using neural networks and partial conventional models, B. Schenker and M. Agarwal data rectification for dynamic processes using artificial neural networks, T.W. Karjala and D.M. Himmelblau applications of neural networks in process dynamics, A.B. Bulsari process modelling for fault detection using neural networks, T. Fujiwara local modelling as a tool for semi-empirical or semi-mechanistic process modelling, B.A. Foss and T.A. Johansen estimation of measurement error variances and process data reconciliation, C. Aldrich and J.S.J. van Deventer process monitoring and visualization using self-organizing maps, O. Simula and J. Kangas an overview of dynamic system control using neural networks, P. Zufiria nonlinear system identification using neural networks - dynamics and instabilities, R. Rico-Martinez et al pattern-based interpretation of on-line process data, J.F. Davis and C.-M. Wang modelling ill-defined behaviour of reacting systems using neural networks, C. Aldrich and J.S.J. van Deventer global vs local networks in identification and control -a case study of neutralization, M.N. Karim and B. Eikens modelling chemical processes using multiresolution representation neural networks, K. Yoda and T. Furuya the videographic characterisation of flotation froths using neural networks, D.W. Moolman et al fuzzy modelling using two connectionist architectures, J. Zhang and A.J. Morris system identification using elman and Jordan networks, D.T. Pham et al time-series prediction with on-line correction of kalman gain - a connectionist approach, A. Dobnikar et al neural networks based control strategies for a continuous polymerisation reactor, N. Watanabe statistical and neural methods in classification and modelling, E.B. Martin et al clustering and statistical techniques in neural networks, V. Venkatasubramanian and R. Rengaswamy.

Local learning in probabilistic networks with hidden variables

Abstract: Probabilistic networks which provide compact descriptions of complex stochastic relationships among several random variables are rapidly becoming the tool of choice for uncertain reasoning in artificial intelligence. We show that networks with fixed structure containing hidden variables can be learned automatically from data using a gradient-descent mechanism similar to that used in neural networks We also extend the method to networks with intensionally represented distributions, including networks with continuous variables and dynamic probabilistic networks Because probabilistic networks provide explicit representations of causal structure human experts can easily contribute pnor knowledge to the training process, thereby significantly improving the learning rate Adaptive probabilistic networks (APNs) may soon compete directly with neural networks as models in computational neuroscience as well as in industrial and financial applications.

Bayesian neural networks and density networks

Abstract: This paper reviews the Bayesian approach to learning in neural networks, then introduces a new adaptive model, the density network. This is a neural network for which target outputs are provided, but the inputs are unspecified. When a probability distribution is placed on the unknown inputs, a latent variable model is defined that is capable of discovering the underlying dimensionality of a data set. A Bayesian learning algorithm for these networks is derived and demonstrated.

Exact Solution for On-Line Learning in Multilayer Neural Networks

Abstract: We present an analytic solution to the problem of on-line gradient-descent learning for two-layer neural networks with an arbitrary number of hidden units in both teacher and student networks.

Artificial Neural Networks: An Introduction to Ann Theory and Practice

Abstract: Artificial neural networks in Neurosciences. This article shows that artificial neural networks are used for confirming the relationships between physiological and cognitive changes. Specifically, we explore the influence of a decrease of neurotransmitters on the behaviour of old people in recognition tasks. This artificial neural network recognizes learned patterns. When we change the threshold of activation in some units, the artificial neural network simulates the experimental results of old people in recognition tasks. However, the main contributions of this paper are the design of an artificial neural network and its operation inspired by the nervous system and the way the inputs are coded and the process of orthogonalization of patterns.

Artificial neural networks for decision support in clinical medicine.

Abstract: Connectionist models such as neural networks are alternatives to linear, parametric statistical methods. Neural networks are computer-based pattern recognition methods with loose similarities with the nervous system. Individual variables of the network, usually called 'neurones', can receive inhibitory and excitatory inputs from other neurones. The networks can define relationships among input data that are not apparent when using other approaches, and they can use these relationships to improve accuracy. Thus, neural nets have substantial power to recognize patterns even in complex datasets. Neural network methodology has outperformed classical statistical methods in cases where the input variables are interrelated. Because clinical measurements usually derive from multiple interrelated systems it is evident that neural networks might be more accurate than classical methods in multivariate analysis of clinical data. This paper reviews the use of neural networks in medical decision support. A short introduction to the basics of neural networks is given, and some practical issues in applying the networks are highlighted. The current use of neural networks in image analysis, signal processing and laboratory medicine is reviewed. It is concluded that neural networks have an important role in image analysis and in signal processing. However, further studies are needed to determine the value of neural networks in the analysis of laboratory data.

Developments in Probabilistic Modelling with Neural Networks — Ensemble Learning

Abstract: Ensemble learning by variational free energy minimization is a framework for statistical inference in which an ensemble of parameter vectors is optimized rather than a single parameter vector. The ensemble approximates the posterior probability distribution of the parameters.

Neural network applications

Abstract: Artificial neural networks, also called neural networks, have been used successfully in many fields including engineering, science and business. This paper presents the implementation of several neural network simulators and their applications in character recognition and other engineering areas. >

Neural Network Initialization

Abstract: Proper initialization is one of the most important prerequisites for fast convergence of feed-forward neural networks like high order and multilayer perceptrons. This publication aims at determining the optimal value of the initial weight variance (or range), which is the principal parameter of random weight initialization methods for both types of neural networks.

Universal learning network and computation of its higher order derivatives

Abstract: In this paper, the universal learning network (ULN) is presented, which models and controls large scale complicated systems such as industrial plants, economics, social and life phenomena. The computing method of higher order derivatives of ULN is derived in order to obtain the learning ability. The basic idea of ULN is that large scale complicated systems can be modeled by the network which consists of nonlinearly operated nodes and branches which may have arbitrary time delays including zero or minus ones. It is shown that the first order derivatives of ULN with sigmoid functions and one sampling time delays correspond to the backpropagation learning algorithm of recurrent neural networks.

The estimation theory and optimization algorithm for the number of hidden units in the higher-order feedforward neural network

Abstract: Estimation theory for the number of the hidden units in the higher-order feedforward neural network has been investigated in this paper and the authors have demonstrated that: for an arbitrary function Y defined on the set S/spl sub/R/sup d/ with (m+1)m/2 elements, the second-order three-layer feedforward neural network with m hidden units can realize function Y sufficiently. With the theories discussed in Kayama et al. (1990), the algorithm for obtaining the optimal number of hidden units has been improved in this paper. Finally, the estimation theory and the developed method are applied to the problems of the prediction of time series and system identification by higher-order neural network, the simulation results show these methods are very effective.

Constructive feedforward neural networks for regression problems : a survey

Abstract: In this paper we review the procedures for constructing feedforward neural networks in regression problems While standard back propagation performs gradient descent only in the weight space of a network with xed topology constructive procedures start with a small network and then grow additional hidden units and weights until a satisfactory solution is found The constructive procedures are categorized according to the resultant network architecture and the learning algorithm for the network weights The Hong Kong University of Science Technology Technical Report Series Department of Computer Science

On-line backpropagation in two-layered neural networks

Abstract: We present an exact analysis of learning a rule by on-line gradient descent in a two-layered neural network with adjustable hidden-to-output weights (backpropagation of error). Results are compared with the training of networks having the same architecture but fixed weights in the second layer.

Constructive Neural Network Learning Algorithms for Multi-Category Pattern Classification

Abstract: Constructive learning algorithms o er an approach for incremental construction of potentially near-minimal neural network architectures for pattern classi cation tasks. Such algorithms help overcome the need for ad-hoc and often inappropriate choice of network topology in the use of algorithms that search for a suitable weight setting in an otherwise a-priori xed network architecture. Several such algorithms proposed in the literature have been shown to converge to zero classi cation errors (under certain assumptions) on a nite, non-contradictory training set in a 2-category classi cation problem. This paper explores multi-category extensions of several constructive neural network learning algorithms for pattern classi cation. In each case, we establish the convergence to zero classi cation errors on a multicategory classi cation task (under certain assumptions). Results of experiments with non linearly separable multi-category data sets demonstrate the feasibility of this approach to multi-category pattern classi cation and also suggest several interesting directions for future research. This research was partially supported by the National Science Foundation grant IRI-9409580 to Vasant Honavar.

ChemNet: A Novel Neural Network Based Method for Graph/Property Mapping

Abstract: ChemNet is a new method for mapping molecular properties. The input of ChemNet consists of twodimensional matrices of variable sizes, instead of the sets of molecular descriptors used by the conventional mapping methods. The two-dimensional matrices considered in this study are graph distance matrices. The diagonal elements of the matrices are atomic properties. ChemNet uses these matrices to form the topology of the artificial neural network. Each molecule of a training set corresponds to a single network configuration. The weighted connections of the networks are adjusted, using the “backprop” procedure. The original background of the method and details of the current realization are presented. Examples of how ChemNet learns topological and physicochemical molecular properties demonstrate the practical use of the method.

Dynamics of On-Line Gradient Descent Learning for Multilayer Neural Networks

Abstract: We consider the problem of on-line gradient descent learning for general two-layer neural networks. An analytic solution is presented and used to investigate the role of the learning rate in controlling the evolution and convergence of the learning process.

Tracking and Recognition of Face Sequences

Abstract: In this work, we address the issue of encoding and recognition of face sequences that arise from continuous head movement. We detect and track a moving head before segmenting face images from on-line camera inputs. We measure temporal changes in the pattern vectors of eigenface projections of successive image frames of a face sequence and introduce the concept of “temporal signature” of a face class. We exploit two different supervised learning algorithms with feedforward and partially recurrent neural networks to learn possible temporal signatures. We discuss our experimental results and draw conclusions.

Design of supervised classifiers using Boolean neural networks

Abstract: In this paper we present two supervised pattern classifiers designed using Boolean neural networks. They are: 1) nearest-to-an-exemplar classifier; and 2) Boolean k-nearest neighbor classifier. The emphasis during the design of these classifiers was on simplicity, robustness, and the ease of hardware implementation. The classifiers use the idea of radius of attraction to achieve their goal. Mathematical analysis of the algorithms presented in the paper is done to prove their feasibility. Both classifiers are tested with well-known binary and continuous feature valued data sets yielding results comparable with those obtained by similar existing classifiers.

A neural network based corner detection method

Abstract: Existing corner detection methods either extract boundaries and search for points having maximum curvature or apply a local operator in parallel to neighborhoods of a gray level picture. The key problem in these methods is the conversion of the gray levels of a pixel into a value reflecting a property of cornerness at that point. A neural network's ability to learn and to adapt together with its inherent parallelism and robustness has made it a natural choice for machine vision applications. This paper presents the application of neural networks to the problem of detecting corners in 2-D images. The performance of the system suggests its robustness and great potential.