A learning rule of neural networks via simultaneous perturbation and its hardware implementation
Summary (1 min read)
1. INTRODUCTION
- The authors show some computer simulations of the proposed learning rule and a comparison between this learning rule, back-propagation method, and a learning rule using the simple perturbation.
- In addition, the authors fabricated an analog neural network circuit using the learning rule.
- The authors describe details of the circuit and show some results by the circuit.
PERTURBATION
- The authors use the following nomenclature in this paper.
- A subscript represents the cell number in the layer.
- On the basis of this estimated first-differential coefficient, the authors can modify the weight vector as with the back-propagation method.
- On the other hand, the authors add the perturbation to all weights simultaneously.
3. SIMULATION RESULTS
- The authors could obtain a neural network learning the 26 characters by using their learning rule.
- Even in this case, the authors require only twice forward operations of the network to obtain the modifying quantities corresponding to all weights.
4.1. Neuron Unit
- In a learning mode, all weight parts in the unit update the weight values in parallel by using the quantity delivered from the learning unit and the sign of the perturbation held in each D-FF.
- Therefore, concurrent modifications of all weights are possible.
5.1. The Exclusive-OR Problem
- Potentially, their leaning rule contains an ability to pass through the local minimum.
- The modifying quantity defined in eqn (3) consists of the first-differential coefficient and an another error as described in eqn (6).
- Thus, the larger c is, the larger the error is, and vice versa if c is smaller.
- From this point of view, the learning rule has a property like the simulated annealing.
- The authors need detailed discussions, analysis and experiences for this point.
5.2. The TCLX Problem
- Figure 15 shows the teaching signals and the observed outputs for this problem.
- This figure shows that the neural network circuit learns the TCLX patterns.
- Also in this problem, the modifications of all weights are performed for a period T/4.
- The operation speed in this figure is approximately 6.0.
Did you find this useful? Give us your feedback
Citations
759 citations
638 citations
570 citations
Additional excerpts
...[647], [655], [670], [679], [682], [693], [698], [699], [702]–...
[...]
...[646], [648]–[653], [655]–[664], [666]–[683], [685]–[699],...
[...]
...[655], [669], [682], [698], [699], [708], [710], [712], [713],...
[...]
378 citations
314 citations
Cites background from "A learning rule of neural networks ..."
...This is a relatively complex operation to be implemented in electronic circuits, and the complications are amplified by imperfections and mismatch in the circuit components....
[...]
References
2,149 citations
Additional excerpts
...Kiefer-Wolfowitz stochastic approximation method (Spall, 1992)....
[...]
1,017 citations
359 citations
"A learning rule of neural networks ..." refers methods in this paper
...Nowadays, we can implement artificial neural networks using several media (De Gloria, 1989; Mead & Ismail, 1989)....
[...]
264 citations
"A learning rule of neural networks ..." refers background or methods in this paper
...Independently, the authors also proposed and fabricated an analog neural network circuit using the same learning rule (Maeda, Yamashita, & Kanata, 1991 ) and investigated a usefulness of this type of learning rule in an inverse problem (Maeda, 1992). However, as pointed out in Maeda, Yamashita, and Kanata (1991), the learning rule using the simple perturbation requires n-times ~ forward operations of the neural network for one modification of all weights....
[...]
...Usually, the learning rule of neural networks via a simple sequential parameter perturbation was proposed and a hardware implementation was reported (Jabri & Flower, 1992)....
[...]
163 citations