Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

This chapter introduces the subject of statistical pattern recognition (SPR). It starts by considering how features are defined and emphasizes that the nearest neighbor algorithm achieves error rates comparable with those of an ideal Bayes’ classifier. The concepts of an optimal number of features, representativeness of the training data, and the need to avoid overfitting to the training data are stressed. The chapter shows that methods such as the support vector machine and artificial neural networks are subject to these same training limitations, although each has its advantages. For neural networks, the multilayer perceptron architecture and back-propagation algorithm are described. The chapter distinguishes between supervised and unsupervised learning, demonstrating the advantages of the latter and showing how methods such as clustering and principal components analysis fit into the SPR framework. The chapter also defines the receiver operating characteristic, which allows an optimum balance between false positives and false negatives to be achieved.

Statistical Pattern Recognition

Guided by brain-like ‘spiking’ computational frameworks, neuromorphic computing—brain-inspired computing for machine intelligence—promises to realize artificial intelligence while reducing the energy requirements of computing platforms. This interdisciplinary field began with the implementation of silicon circuits for biological neural routines, but has evolved to encompass the hardware implementation of algorithms with spike-based encoding and event-driven representations. Here we provide an overview of the developments in neuromorphic computing for both algorithms and hardware and highlight the fundamentals of learning and hardware frameworks. We discuss the main challenges and the future prospects of neuromorphic computing, with emphasis on algorithm–hardware codesign. The authors review the advantages and future prospects of neuromorphic computing, a multidisciplinary engineering concept for energy-efficient artificial intelligence with brain-inspired functionality.

Towards spike-based machine intelligence with neuromorphic computing.

Deep learning in spiking neural networks

Spiking neural networks (SNNs) are nature's versatile solution to fault-tolerant, energy-efficient signal processing. To translate these benefits into hardware, a growing number of neuromorphic spiking NN processors have attempted to emulate biological NNs. These developments have created an imminent need for methods and tools that enable such systems to solve real-world signal processing problems. Like conventional NNs, SNNs can be trained on real, domain-specific data; however, their training requires the overcoming of a number of challenges linked to their binary and dynamical nature. This article elucidates step-by-step the problems typically encountered when training SNNs and guides the reader through the key concepts of synaptic plasticity and data-driven learning in the spiking setting. Accordingly, it gives an overview of existing approaches and provides an introduction to surrogate gradient (SG) methods, specifically, as a particularly flexible and efficient method to overcome the aforementioned challenges.

Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks

BP-STDP: Approximating backpropagation using spike timing dependent plasticity

The problem of training spiking neural networks (SNNs) is a necessary precondition to understanding computations within the brain, a field still in its infancy. Previous work has shown that supervised learning in multi-layer SNNs enables bio-inspired networks to recognize patterns of stimuli through hierarchical feature acquisition. Although gradient descent has shown impressive performance in multi-layer (and deep) SNNs, it is generally not considered biologically plausible and is also computationally expensive. This paper proposes a novel supervised learning approach based on an event-based spike-timing-dependent plasticity (STDP) rule embedded in a network of integrate-and-fire (IF) neurons. The proposed temporally local learning rule follows the backpropagation weight change updates applied at each time step. This approach enjoys benefits of both accurate gradient descent and temporally local, efficient STDP. Thus, this method is able to address some open questions regarding accurate and efficient computations that occur in the brain. The experimental results on the XOR problem, the Iris data, and the MNIST dataset demonstrate that the proposed SNN performs as successfully as the traditional NNs. Our approach also compares favorably with the state-of-the-art multi-layer SNNs.

BP-STDP: Approximating Backpropagation using Spike Timing Dependent Plasticity

We propose a novel data-driven machine learning method using long short-term memory (LSTM)-based multi-stage forecasting for influenza forecasting. The novel aspects of the method include the following: 1) the introduction of LSTM method to capture the temporal dynamics of seasonal flu and 2) a technique to capture the influence of external variables that includes the geographical proximity and climatic variables such as humidity, temperature, precipitation, and sun exposure. The proposed model is compared against two state-of-the-art techniques using two publicly available datasets. Our proposed method performs better than the existing well-known influenza forecasting methods. The results offer a promising direction in terms of both using the data-driven forecasting methods and capturing the influence of spatio-temporal and environmental factors to improve influenza forecasting.

/pdf/a-novel-data-driven-model-for-real-time-influenza-2k3gnivc0l.pdf

A Novel Data-Driven Model for Real-Time Influenza Forecasting

Spiking neural networks (SNNs) have advantages over traditional, non-spiking networks with respect to biorealism, potential for low-power hardware implementations, and theoretical computing power. However, in practice, spiking networks with multi-layer learning have proven difficult to train. This paper explores a novel, bio-inspired spiking convolutional neural network (CNN) that is trained in a greedy, layer-wise fashion. The spiking CNN consists of a convolutional/pooling layer followed by a feature discovery layer, both of which undergo bio-inspired learning. Kernels for the convolutional layer are trained using a sparse, spiking auto-encoder representing primary visual features. The feature discovery layer uses a probabilistic spike-timing-dependent plasticity (STDP) learning rule. This layer represents complex visual features using WTA-thresholded, leaky, integrate-and-fire (LIF) neurons. The new model is evaluated on the MNIST digit dataset using clean and noisy images. Intermediate results show that the convolutional layer is stack-admissible, enabling it to support a multi-layer learning architecture. The recognition performance for clean images is above 98%. This performance is accounted for by the independent and informative visual features extracted in a hierarchy of convolutional and feature discovery layers. The performance loss for recognizing the noisy images is in the range 0.1% to 8.5%. This level of performance loss indicates that the network is robust to additive noise.

Amirhossein Tavanaei

Papers

Deep learning in spiking neural networks

BP-STDP: Approximating backpropagation using spike timing dependent plasticity

BP-STDP: Approximating Backpropagation using Spike Timing Dependent Plasticity

A Novel Data-Driven Model for Real-Time Influenza Forecasting

Multi-layer unsupervised learning in a spiking convolutional neural network