Just storing the Hessian H (the matrix of second derivatives δ2E/δwiδ wj of the error E with respect to each pair of weights) of a large neural network is difficult. Since a common use of a large matrix like H is to compute its product with various vectors, we derive a technique that directly calculates Hv, where v is an arbitrary vector. To calculate Hv, we first define a differential operator Rv{f(w)} = (δ/δr)f(w + rv)|r=0, note that Rv{∇w} = Hv and Rv{w} = v, and then apply Rv{·} to the equations used to compute ∇w. The result is an exact and numerically stable procedure for computing Hv, which takes about as much computation, and is about as local, as a gradient evaluation. We then apply the technique to a one pass gradient calculation algorithm (backpropagation), a relaxation gradient calculation algorithm (recurrent backpropagation), and two stochastic gradient calculation algorithms (Boltzmann machines and weight perturbation). Finally, we show that this technique can be used at the heart of many iterative techniques for computing various properties of H, obviating any need to calculate the full Hessian.

/pdf/fast-exact-multiplication-by-the-hessian-1cfxtdsbcs.pdf

Fast exact multiplication by the Hessian

Neuromorphic computing has come to refer to a variety of brain-inspired computers, devices, and models that contrast the pervasive von Neumann computer architecture This biologically inspired approach has created highly connected synthetic neurons and synapses that can be used to model neuroscience theories as well as solve challenging machine learning problems The promise of the technology is to create a brain-like ability to learn and adapt, but the technical challenges are significant, starting with an accurate neuroscience model of how the brain works, to finding materials and engineering breakthroughs to build devices to support these models, to creating a programming framework so the systems can learn, to creating applications with brain-like capabilities In this work, we provide a comprehensive survey of the research and motivations for neuromorphic computing over its history We begin with a 35-year review of the motivations and drivers of neuromorphic computing, then look at the major research areas of the field, which we define as neuro-inspired models, algorithms and learning approaches, hardware and devices, supporting systems, and finally applications We conclude with a broad discussion on the major research topics that need to be addressed in the coming years to see the promise of neuromorphic computing fulfilled The goals of this work are to provide an exhaustive review of the research conducted in neuromorphic computing since the inception of the term, and to motivate further work by illuminating gaps in the field where new research is needed

A Survey of Neuromorphic Computing and Neural Networks in Hardware.

A new optimization technique, stochastic parallel-gradient descent, is applied for high-resolution adaptive wave-front correction. A performance criterion for parallel-perturbation-based algorithms is introduced and applied to optimize adaptive system architecture. We present numerical simulation results for an adaptive imaging system based on the stochastic parallel-perturbation technique, along with experimental results obtained for a white-light adaptive imaging system with 37 control channels. An adaptive system with a self-organized (adaptive) control channel hierarchy is introduced and analyzed.

Stochastic parallel-gradient-descent technique for high-resolution wave-front phase-distortion correction

Machine learning develops rapidly, which has made many theoretical breakthroughs and is widely applied in various fields. Optimization, as an important part of machine learning, has attracted much attention of researchers. With the exponential growth of data amount and the increase of model complexity, optimization methods in machine learning face more and more challenges. A lot of work on solving optimization problems or improving optimization methods in machine learning has been proposed successively. The systematic retrospect and summary of the optimization methods from the perspective of machine learning are of great significance, which can offer guidance for both developments of optimization and machine learning research. In this article, we first describe the optimization problems in machine learning. Then, we introduce the principles and progresses of commonly used optimization methods. Finally, we explore and give some challenges and open problems for the optimization in machine learning.

A Survey of Optimization Methods From a Machine Learning Perspective

neural and fuzzy logic control of drives and power systems neural and fuzzy logic control of drives and power systems neural and fuzzy logic control of drives and power systems neural and fuzzy logic control of drives and power systems neural and fuzzy logic control of drives power systems neural and fuzzy logic control of drives and power systems neural and fuzzy logic control of drives and power systems chapter 8 fuzzy logic and neural network applications in fuzzy logic applications to power electronics and drives expert system, fuzzy logic, and neural network speed control of dc motor using fuzzy logic technique dynamic fuzzy-neural network controller for induction neural and fuzzy logic control of drives and power systems advanced fuzzy systems design and applications khbd adaptive neural fuzzy inference systems controller for induction motor drive using fuzzy logic citeseerx embedded fuzzy controller for industrial applications direct torque control of induction motor using fuzzy logic application of learning methodologies in control of power fuzzy logic application for intelligent control of a speed control of dc motor using fuzzy pid controller application of fuzzy logic in engineering problems lnai 3949 dsp based fuzzy-neural speed tracking control a servo motor drives based on hybrid genetic algorithm gvpce(a ) m.tech. power electronics & drives 2013 neural fuzzy rule – based model reference adaptive control for artificial neural network and logic control for hvdc systems fuzzy logic control of maximum power point tracking for controlling speed of dc motor with fuzzy controller in speed control of brushless dc motor using fuzzy ijdacr design and performance evaluation of a fuzzy-logic-based maximum power point tracking of pv system using anfis power electronics and motor drives elsevier

/pdf/neural-and-fuzzy-logic-control-of-drives-and-power-systems-2gnm35onla.pdf

Neural and Fuzzy Logic Control of Drives and Power Systems

Typical methods for gradient descent in neural network learning involve calculation of derivatives based on a detailed knowledge of the network model. This requires extensive, time consuming calculations for each pattern presentation and high precision that makes it difficult to implement in VLSI. We present here a perturbation technique that measures, not calculates, the gradient. Since the technique uses the actual network as a measuring device, errors in modeling neuron activation and synaptic weights do not cause errors in gradient descent. The method is parallel in nature and easy to implement in VLSI. We describe the theory of such an algorithm, an analysis of its domain of applicability, some simulations using it and an outline of a hardware implementation.

/pdf/a-parallel-gradient-descent-method-for-learning-in-analog-21vm4dnag3.pdf

A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks

We report learning measurements from a system composed of a cascadable learning chip, data generators and analyzers for training pattern presentation, and an X-windows based software interface. The 32 neuron learning chip has 496 adaptive synapses and can perform Boltzmann and mean-field learning using separate noise and gain controls. We have used this system to do learning experiments on the parity and replication problem. The system settling time limits the learning speed to about 100,000 patterns per second roughly independent of system size.

/pdf/experimental-evaluation-of-learning-in-a-neural-microsystem-4yfojctrx8.pdf

Experimental Evaluation of Learning in a Neural Microsystem

Feedback connections are required so that the teacher signal on the output neurons can modify weights during supervised learning. Relaxation methods are needed for learning static patterns with full-time feedback connections. Feedback network learning techniques have not achieved wide popularity because of the still greater computational efficiency of back-propagation. We show by simulation that relaxation networks of the kind we are implementing in VLSI are capable of learning large problems just like back-propagation networks. A microchip incorporates deterministic mean-field theory learning as well as stochastic Boltzmann learning. A multiple-chip electronic system implementing these networks will make high-speed parallel learning in them feasible in the future.

https://papers.nips.cc/paper/367-relaxation-networks-for-large-supervised-learning-problems.pdf

Relaxation Networks for Large Supervised Learning Problems

A channel equalizer is formed using a self-learning neural network. During a training period, the neural network is taught the channel response function. The network is then used to equalize distortions introduced into signals by the channel. The neural network may be a Boltzmann Machine type of neural network comprising neurons arranged in an input layer, a hidden layer, and an output layer. The neurons are interconnected by bidirectional symmetric weighted synapses. Each neuron is preferably implemented by an analog integrated circuit. Direct communication between the input and output layers helps in faster channel acquisition. The scheme can very easily be extended to multilevel and multisymbol modulation schemes such as QAM and PSK.

Adaptive equalizer using self-learning neural network

A neuron for use in a self-learning neural network comprises a current input node at which a plurality of synaptic input currents are summed using Kirchoff's current law. The summed input currents are normalized using a coarse gain current normalizer. The normalized summed inputs current is then converted to a voltage using a current to voltage converter. This voltage is then amplified by a gain controlled cascode output amplifier. Gain control inputs are provided in the output amplifier so that the neuron can be settled by the Mean Field Approximation. A noise input stage is also connected to the output amplifier so that the neuron can be settled using simulated annealing. The resulting neuron is a variable gain, bi-directional current transimpedance neuron with a controllable noise input.

Anthony Jayakumar

Papers

A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks

Experimental Evaluation of Learning in a Neural Microsystem

Relaxation Networks for Large Supervised Learning Problems

Adaptive equalizer using self-learning neural network

Neuron for use in self-learning neural network