Just storing the Hessian H (the matrix of second derivatives δ2E/δwiδ wj of the error E with respect to each pair of weights) of a large neural network is difficult. Since a common use of a large matrix like H is to compute its product with various vectors, we derive a technique that directly calculates Hv, where v is an arbitrary vector. To calculate Hv, we first define a differential operator Rv{f(w)} = (δ/δr)f(w + rv)|r=0, note that Rv{∇w} = Hv and Rv{w} = v, and then apply Rv{·} to the equations used to compute ∇w. The result is an exact and numerically stable procedure for computing Hv, which takes about as much computation, and is about as local, as a gradient evaluation. We then apply the technique to a one pass gradient calculation algorithm (backpropagation), a relaxation gradient calculation algorithm (recurrent backpropagation), and two stochastic gradient calculation algorithms (Boltzmann machines and weight perturbation). Finally, we show that this technique can be used at the heart of many iterative techniques for computing various properties of H, obviating any need to calculate the full Hessian.

/pdf/fast-exact-multiplication-by-the-hessian-1cfxtdsbcs.pdf

Fast exact multiplication by the Hessian

Neuromorphic computing has come to refer to a variety of brain-inspired computers, devices, and models that contrast the pervasive von Neumann computer architecture This biologically inspired approach has created highly connected synthetic neurons and synapses that can be used to model neuroscience theories as well as solve challenging machine learning problems The promise of the technology is to create a brain-like ability to learn and adapt, but the technical challenges are significant, starting with an accurate neuroscience model of how the brain works, to finding materials and engineering breakthroughs to build devices to support these models, to creating a programming framework so the systems can learn, to creating applications with brain-like capabilities In this work, we provide a comprehensive survey of the research and motivations for neuromorphic computing over its history We begin with a 35-year review of the motivations and drivers of neuromorphic computing, then look at the major research areas of the field, which we define as neuro-inspired models, algorithms and learning approaches, hardware and devices, supporting systems, and finally applications We conclude with a broad discussion on the major research topics that need to be addressed in the coming years to see the promise of neuromorphic computing fulfilled The goals of this work are to provide an exhaustive review of the research conducted in neuromorphic computing since the inception of the term, and to motivate further work by illuminating gaps in the field where new research is needed

A Survey of Neuromorphic Computing and Neural Networks in Hardware.

A system for computer-based storing of information about a current state so that later recall of the information can augment human memories. In particular, when information about a current event of interest is to be stored, a variety of current state information of different types (e.g., video, audio, and textual information) about the environment and about a user can be acquired via sensors and other input devices. The variety of state information can then be associated together as a group and stored for later retrieval. Other information can also be associated with the group, such as one or more recall tags that facilitate later retrieval of the group, or one or more annotations to provide contextual information when the other state information is later retrieved and presented to the user. When information about a past event is to be recalled, one or more identifying recall tags can be received that are used to identify one or more state information groups that match the identifying tags. Some or all of the previously-acquired state information for the identified state information groups can then be presented to the user on appropriate output devices. Other information, such as annotations, can also be presented to the user in order to describe the state information and thus assist the user's recollection of the previous state when the information was stored.

Storing and recalling information to augment human memories

A new optimization technique, stochastic parallel-gradient descent, is applied for high-resolution adaptive wave-front correction. A performance criterion for parallel-perturbation-based algorithms is introduced and applied to optimize adaptive system architecture. We present numerical simulation results for an adaptive imaging system based on the stochastic parallel-perturbation technique, along with experimental results obtained for a white-light adaptive imaging system with 37 control channels. An adaptive system with a self-organized (adaptive) control channel hierarchy is introduced and analyzed.

Stochastic parallel-gradient-descent technique for high-resolution wave-front phase-distortion correction

A review is presented of ATR (automatic target recognition), and some of the highlights of neural network technology developments that have the potential for making a significant impact on ATR are presented. In particular, neural network technology developments in the areas of collective computation, learning algorithms, expert systems, and neurocomputer hardware could provide crucial tools for developing improved algorithms and computational hardware for ATR. The discussion covers previous ATR system efforts. ATR issues and needs, early vision and collective computation, learning and adaptation for ATR, feature extraction, higher vision and expert systems, and neurocomputer hardware. >

Survey of neural network technology for automatic target recognition

of the Disclosure A neuron network which achieves learning by means of a modified Boltzmann algorithm. The network may comprise interconnected input, hidden and output layers of neurons, the neurons being " on-off' or threshold electronic symmetrically connected by means of adjustable-weight synapse pairs. The synapses comprise the source-drain circuits of a plurality of paralleled FET's which differ in resistance or conductance in a binary sequence. The synapses are controlled by the output of an Up-Down counter, the reading of which is controlled by the results of a correlation of the states of the two neuron connected by the synapse pairs following the application of a set of plus and minus training signals to selected neurons of said network. A noise generator comprising a thermal noise source is provided for each neuron for the purpose of simulated annealing of the network.

Neuromorphic learning networks

Typical methods for gradient descent in neural network learning involve calculation of derivatives based on a detailed knowledge of the network model. This requires extensive, time consuming calculations for each pattern presentation and high precision that makes it difficult to implement in VLSI. We present here a perturbation technique that measures, not calculates, the gradient. Since the technique uses the actual network as a measuring device, errors in modeling neuron activation and synaptic weights do not cause errors in gradient descent. The method is parallel in nature and easy to implement in VLSI. We describe the theory of such an algorithm, an analysis of its domain of applicability, some simulations using it and an outline of a hardware implementation.

/pdf/a-parallel-gradient-descent-method-for-learning-in-analog-21vm4dnag3.pdf

A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks

We have fabricated a test chip in 2 micron CMOS that can perform supervised learning in a manner similar to the Boltzmann machine. Patterns can be presented to it at 100,000 per second. The chip learns to solve the XOR problem in a few milliseconds. We also have demonstrated the capability to do unsupervised competitive learning with it. The functions of the chip components are examined and the performance is assessed.

Performance of a stochastic learning microchip

We describe a family of learning algorithms that operate on a recurrent, symmetrically connected, neuromorphic network that, like the Boltzmann machine, settles in the presence of noise. These networks learn by modifying synaptic connection strengths on the basis of correlations seen locally by each synapse. We describe a version of the supervised learning algorithm for a network with analog activation functions. We also demonstrate unsupervised competitive learning with this approach, where weight saturation and decay play an important role, and describe preliminary experiments in reinforcement learning, where noise is used in the search procedure. We identify the above described phenomena as elements that can unify learning techniques at a physical microscopic level.

These algorithms were chosen for ease of implementation in vlsi. We have designed a CMOS test chip in 2 micron rules that can speed up the learning about a millionfold over an equivalent simulation on a VAX 11/780. The speedup is due to parallel analog computation for summing and multiplying weights and activations, and the use of physical processes for generating random noise. The components of the test chip are a noise amplifier, a neuron amplifier, and a 300 transistor adaptive synapse, each of which is separately testable. These components are also integrated into a 6 neuron and 15 synapse network. Finally, we point out techniques for reducing the area of the electronic correlational synapse both in technology and design and show how the algorithms we study can be implemented naturally in electronic systems.

/pdf/stochastic-learning-networks-and-their-electronic-60w74rr6bl.pdf

Stochastic Learning Networks and their Electronic Implementation

We report learning measurements from a system composed of a cascadable learning chip, data generators and analyzers for training pattern presentation, and an X-windows based software interface. The 32 neuron learning chip has 496 adaptive synapses and can perform Boltzmann and mean-field learning using separate noise and gain controls. We have used this system to do learning experiments on the parity and replication problem. The system settling time limits the learning speed to about 100,000 patterns per second roughly independent of system size.

/pdf/experimental-evaluation-of-learning-in-a-neural-microsystem-4yfojctrx8.pdf

Joshua Alspector

Papers

Neuromorphic learning networks

A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks

Performance of a stochastic learning microchip

Stochastic Learning Networks and their Electronic Implementation

Experimental Evaluation of Learning in a Neural Microsystem