A Circuit-Based Learning Architecture for Multilayer Neural Networks With Memristor Bridge Synapses
01 Jan 2015-IEEE Transactions on Circuits and Systems I-regular Papers (IEEE)-Vol. 62, Iss: 1, pp 215-223
TL;DR: It is shown that a circuit-based learning using RWC is two orders faster than its software counterpart, which is a first of its kind demonstrating successful circuit- based learning for multilayer neural network built with memristors.
Abstract: Memristor-based circuit architecture for multilayer neural networks is proposed. It is a first of its kind demonstrating successful circuit-based learning for multilayer neural network built with memristors. Though back-propagation algorithm is a powerful learning scheme for multilayer neural networks, its hardware implementation is very difficult due to complexities of the neural synapses and the operations involved in the learning algorithm. In this paper, the circuit of a multilayer neural network is designed with memristor bridge synapses and the learning is realized with a simple learning algorithm called Random Weight Change (RWC). Though RWC algorithm requires more iterations than back-propagation algorithm, we show that a circuit-based learning using RWC is two orders faster than its software counterpart. The method to build a multilayer neural network using memristor bridge synapses and a circuit-based learning architecture of RWC algorithm is proposed. Comparison between software-based and memristor circuit-based learning are presented via simulations.
Citations
More filters
Posted Content•
TL;DR: An exhaustive review of the research conducted in neuromorphic computing since the inception of the term is provided to motivate further work by illuminating gaps in the field where new research is needed.
Abstract: Neuromorphic computing has come to refer to a variety of brain-inspired computers, devices, and models that contrast the pervasive von Neumann computer architecture This biologically inspired approach has created highly connected synthetic neurons and synapses that can be used to model neuroscience theories as well as solve challenging machine learning problems The promise of the technology is to create a brain-like ability to learn and adapt, but the technical challenges are significant, starting with an accurate neuroscience model of how the brain works, to finding materials and engineering breakthroughs to build devices to support these models, to creating a programming framework so the systems can learn, to creating applications with brain-like capabilities In this work, we provide a comprehensive survey of the research and motivations for neuromorphic computing over its history We begin with a 35-year review of the motivations and drivers of neuromorphic computing, then look at the major research areas of the field, which we define as neuro-inspired models, algorithms and learning approaches, hardware and devices, supporting systems, and finally applications We conclude with a broad discussion on the major research topics that need to be addressed in the coming years to see the promise of neuromorphic computing fulfilled The goals of this work are to provide an exhaustive review of the research conducted in neuromorphic computing since the inception of the term, and to motivate further work by illuminating gaps in the field where new research is needed
570 citations
Additional excerpts
...recognition or detection [582], [1067], [1204], [1284], [1885],...
[...]
...[1279], [1527], [1784], [1885], [2550], [2551], social learning...
[...]
TL;DR: This paper provides the review of neuromorphic CMOS-memristive architectures that can be integrated into edge computing devices and discusses why the neuromorphic architectures are useful for edge devices and shows the advantages, drawbacks, and open problems in the field of neuromemristive circuits for edge computing.
Abstract: The volume, veracity, variability, and velocity of data produced from the ever increasing network of sensors connected to Internet pose challenges for power management, scalability, and sustainability of cloud computing infrastructure. Increasing the data processing capability of edge computing devices at lower power requirements can reduce several overheads for cloud computing solutions. This paper provides the review of neuromorphic CMOS-memristive architectures that can be integrated into edge computing devices. We discuss why the neuromorphic architectures are useful for edge devices and show the advantages, drawbacks, and open problems in the field of neuromemristive circuits for edge computing.
201 citations
Cites background or methods from "A Circuit-Based Learning Architectu..."
...In this neuron, the voltage weighted by the memristor bridge synapses is converted to the current using differential amplifiers [23]....
[...]
...This circuit is used in various neural network architectures [48], [23]....
[...]
...2 (f) was tested in various neural network architectures and applications [22], [23]....
[...]
...Memristive synapses: (a) 1M synapse in a crossbar array, (b) 2M synapses [18], (c) 2M1R synapse [19], (d) 1T1M synapses [20], [13], (e) 2T2M synapses [21], (f) 4M synapse [22], [23] and 5M synapse [24]....
[...]
...Neuron cells: (a) Modified I&F neuron [32]; variations of neuron models based on summing amplifier and comparator: (b) [33], [34], (c) [35], (d) [36], (e) [37] and (f) [26]; (g) neuron models with sigmoid activation function [38], (h) neuron model for memristor-bridge architectures [22], [23]; (i) stochastic neuron [39]; and (j) HTM SP neuron [6]....
[...]
TL;DR: A novel circuit for Memristor-based multilayer neural networks is presented, which can use a single memristor array to realize both the plus and minus weight of the neural synapses.
Abstract: Memristors are promising components for applications in nonvolatile memory, logic circuits, and neuromorphic computing. In this paper, a novel circuit for memristor-based multilayer neural networks is presented, which can use a single memristor array to realize both the plus and minus weight of the neural synapses. In addition, memristor-based switches are utilized during the learning process to update the weight of the memristor-based synapses. Moreover, an adaptive back propagation algorithm suitable for the proposed memristor-based multilayer neural network is applied to train the neural networks and perform the XOR function and character recognition. Another highlight of this paper is that the robustness of the proposed memristor-based multilayer neural network exhibits higher recognition rates and fewer cycles as compared with other multilayer neural networks.
163 citations
Cites methods from "A Circuit-Based Learning Architectu..."
...A memristor bridge synapse-based neural network and learning are proposed in [14]–[16], which implement multilayer neural networks (MNN) trained by a back propagation (BP) algorithm, and the synapse weight updates are performed by a host computer....
[...]
...The proposed memristor-based synaptic crossbar circuit uses fewer memristors and no transistors as compared with the synaptic circuits discribed in [11], [12], [14]-[16], [18]-[20], [22], [23], [25], [30], and [31]....
[...]
TL;DR: The fabricated three-dimensional vertical ferroelectric tunneling junction synapse (FTJS) exhibits high integration density and excellent performances, such as analog-like conductance transition under a training scheme, low energy consumption of synaptic weight update and good repeatability.
Abstract: Brain-inspired neuromorphic computing has shown great promise beyond the conventional Boolean logic. Nanoscale electronic synapses, which have stringent demands for integration density, dynamic range, energy consumption, etc., are key computational elements of the brain-inspired neuromorphic system. Ferroelectric tunneling junctions have been shown to be ideal candidates to realize the functions of electronic synapses due to their ultra-low energy consumption and the nature of ferroelectric tunneling. Here, we report a new electronic synapse based on a three-dimensional vertical Hf0.5Zr0.5O2-based ferroelectric tunneling junction that meets the full functions of biological synapses. The fabricated three-dimensional vertical ferroelectric tunneling junction synapse (FTJS) exhibits high integration density and excellent performances, such as analog-like conductance transition under a training scheme, low energy consumption of synaptic weight update (1.8 pJ per spike) and good repeatability (>103 cycles). In addition, the implementation of pattern training in hardware with strong tolerance to input faults and variations is also illustrated in the 3D vertical FTJS array. Furthermore, pattern classification and recognition are achieved, and these results demonstrate that the Hf0.5Zr0.5O2-based FTJS has high potential to be an ideal electronic component for neuromorphic system applications.
140 citations
TL;DR: A novel memristive multilayer CNN (Mm-CNN) model is presented along with its performance analysis and applications, which has several merits, such as compactness, nonvolatility, versatility, and programmability of synaptic weights.
Abstract: The memristor has been extensively studied in electrical engineering and biological sciences as a means to compactly implement the synaptic function in neural networks. The cellular neural network (CNN) is one of the most implementable artificial neural network models and capable of massively parallel analog processing. In this paper, a novel memristive multilayer CNN (Mm-CNN) model is presented along with its performance analysis and applications. In this new CNN design, the memristor crossbar circuit acts as the synapse, which realizes one signed synaptic weight with a pair of memristors and performs the synaptic weighting compactly and linearly. Moreover, the complex weighted summation is executed in an efficient way with a proper design of Mm-CNN cell circuits. The proposed Mm-CNN has several merits, such as compactness, nonvolatility, versatility, and programmability of synaptic weights. Its performance in several image processing applications is illustrated through simulations.
115 citations
Cites background from "A Circuit-Based Learning Architectu..."
...These approaches, however, suffer from the problems of difficult real-time weight updating, power-consuming weight storage, and nonlinearity in synaptic weighting, respectively [21]....
[...]
References
More filters
TL;DR: It is shown, using a simple analytical example, that memristance arises naturally in nanoscale systems in which solid-state electronic and ionic transport are coupled under an external bias voltage.
Abstract: Anyone who ever took an electronics laboratory class will be familiar with the fundamental passive circuit elements: the resistor, the capacitor and the inductor. However, in 1971 Leon Chua reasoned from symmetry arguments that there should be a fourth fundamental element, which he called a memristor (short for memory resistor). Although he showed that such an element has many interesting and valuable circuit properties, until now no one has presented either a useful physical model or an example of a memristor. Here we show, using a simple analytical example, that memristance arises naturally in nanoscale systems in which solid-state electronic and ionic transport are coupled under an external bias voltage. These results serve as the foundation for understanding a wide range of hysteretic current-voltage behaviour observed in many nanoscale electronic devices that involve the motion of charged atomic or molecular species, in particular certain titanium dioxide cross-point switches.
8,971 citations
TL;DR: A nanoscale silicon-based memristor device is experimentally demonstrated and it is shown that a hybrid system composed of complementary metal-oxide semiconductor neurons and Memristor synapses can support important synaptic functions such as spike timing dependent plasticity.
Abstract: A memristor is a two-terminal electronic device whose conductance can be precisely modulated by charge or flux through it. Here we experimentally demonstrate a nanoscale silicon-based memristor device and show that a hybrid system composed of complementary metal−oxide semiconductor neurons and memristor synapses can support important synaptic functions such as spike timing dependent plasticity. Using memristors as synapses in neuromorphic circuits can potentially offer both high connectivity and high density required for efficient computing.
3,650 citations
TL;DR: The memristor is a 2-terminal nonvolatile memory device that exhibits a pinched hysteresis loop confined to the first and third quadrants of the v-i plane whose contour shape in general changes with both the amplitude and frequency of any periodic sine-wave-like input voltage source, or current source as mentioned in this paper.
Abstract: All 2-terminal non-volatile memory devices based on resistance switching are memristors, regardless of the device material and physical operating mechanisms. They all exhibit a distinctive “fingerprint” characterized by a pinched hysteresis loop confined to the first and the third quadrants of the v–i plane whose contour shape in general changes with both the amplitude and frequency of any periodic “sine-wave-like” input voltage source, or current source. In particular, the pinched hysteresis loop shrinks and tends to a straight line as frequency increases. Though numerous examples of voltage vs. current pinched hysteresis loops have been published in many unrelated fields, such as biology, chemistry, physics, etc., and observed from many unrelated phenomena, such as gas discharge arcs, mercury lamps, power conversion devices, earthquake conductance variations, etc., we restrict our examples in this tutorial to solid-state and/or nano devices where copious examples of published pinched hysteresis loops abound. In particular, we sampled arbitrarily, one example from each year between the years 2000 and 2010, to demonstrate that the memristor is a device that does not depend on any particular material, or physical mechanism. For example, we have shown that spin-transfer magnetic tunnel junctions are examples of memristors. We have also demonstrated that both bipolar and unipolar resistance switching devices are memristors.
1,208 citations
01 Jan 2019
TL;DR: The goal of this tutorial is to introduce some fundamental circuit-theoretic concepts and properties of the memristor that are relevant to the analysis and design of non-volatile nano memories where binary bits are stored as resistances manifested by the Memristor’s continuum of equilibrium states.
Abstract: All 2-terminal non-volatile memory devices based on resistance switching are memristors, regardless of the device material and physical operating mechanisms. They all exhibit a distinctive “fingerprint” characterized by a pinched hysteresis loop confined to the first and the third quadrants of the v–i plane whose contour shape in general changes with both the amplitude and frequency of any periodic “sine-wave-like” input voltage source, or current source. In particular, the pinched hysteresis loop shrinks and tends to a straight line as frequency increases. Though numerous examples of voltage vs. current pinched hysteresis loops have been published in many unrelated fields, such as biology, chemistry, physics, etc., and observed from many unrelated phenomena, such as gas discharge arcs, mercury lamps, power conversion devices, earthquake conductance variations, etc., we restrict our examples in this tutorial to solid-state and/or nano devices where copious examples of published pinched hysteresis loops abound. In particular, we sampled arbitrarily, one example from each year between the years 2000 and 2010, to demonstrate that the memristor is a device that does not depend on any particular material, or physical mechanism. For example, we have shown that spin-transfer magnetic tunnel junctions are examples of memristors. We have also demonstrated that both bipolar and unipolar resistance switching devices are memristors.
1,097 citations
"A Circuit-Based Learning Architectu..." refers background in this paper
...Many different devices have since been identified as memristors [5]....
[...]
01 Jan 1989
TL;DR: The authors report the analog storage and multiply characteristics of a new floating-gate synapse and further discuss the architecture of a neural network which uses this synapse cell, using 1- mu m CMOS EEPROM technology.
Abstract: The use of floating-gate nonvolatile memory technology for analog storage of connection strengths, or weights, has previously been proposed and demonstrated. The authors report the analog storage and multiply characteristics of a new floating-gate synapse and further discuss the architecture of a neural network which uses this synapse cell. In the architecture described 8192 synapses are used to interconnect 64 neurons fully and to connect the 64 neurons to each of 64 inputs. Each synapse in the network multiplies a signed analog voltage by a stored weight and generates a differential current proportional to the product. Differential currents are summed on a pair of bit lines and transferred through a sigmoid function, appearing at the neuron output as an analog voltage. Input and output levels are compatible for ease in cascade-connecting these devices into multilayer networks. The width and height of weight-change pulses are calculated. The synapse cell size is 2009 mu m/sup 2/ using 1- mu m CMOS EEPROM technology. >
394 citations
"A Circuit-Based Learning Architectu..." refers background in this paper
...In analog hardware implementations the weights are usually stored in resistors [1], capacitors [2], and floating gate transistors [3]....
[...]