Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

We derive a new self-organizing learning algorithm that maximizes the information transferred in a network of nonlinear units. The algorithm does not assume any knowledge of the input distributions, and is defined here for the zero-noise limit. Under these conditions, information maximization has extra properties not found in the linear case (Linsker 1989). The nonlinearities in the transfer function are able to pick up higher-order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation. This enables the network to separate statistically independent components in the inputs: a higher-order generalization of principal components analysis. We apply the network to the source separation (or cocktail party) problem, successfully separating unknown mixtures of up to 10 speakers. We also show that a variant on the network architecture is able to perform blind deconvolution (cancellation of unknown echoes and reverberation in a speech signal). Finally, we derive dependencies of information transfer on time delays. We suggest that information maximization provides a unifying framework for problems in "blind" signal processing.

/pdf/an-information-maximization-approach-to-blind-separation-and-1cnlj7ibol.pdf

An information-maximization approach to blind separation and blind deconvolution

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

We describe the design of Kaldi, a free, open-source toolkit for speech recognition research. Kaldi provides a speech recognition system based on finite-state automata (using the freely available OpenFst), together with detailed documentation and a comprehensive set of scripts for building complete recognition systems. Kaldi is written is C++, and the core library supports modeling of arbitrary phonetic-context sizes, acoustic modeling with subspace Gaussian mixture models (SGMM) as well as standard Gaussian mixture models, together with all commonly used linear and affine transforms. Kaldi is released under the Apache License v2.0, which is highly nonrestrictive, making it suitable for a wide community of users.

/pdf/the-kaldi-speech-recognition-toolkit-3g790z933t.pdf

The Kaldi Speech Recognition Toolkit

Inspired by the brain’s structure, we have developed an efficient, scalable, and flexible non–von Neumann architecture that leverages contemporary silicon technology. To demonstrate, we built a 5.4-billion-transistor chip with 4096 neurosynaptic cores interconnected via an intrachip network that integrates 1 million programmable spiking neurons and 256 million configurable synapses. Chips can be tiled in two dimensions via an interchip communication interface, seamlessly scaling the architecture to a cortexlike sheet of arbitrary size. The architecture is well suited to many applications that use complex neural networks in real time, for example, multiobject detection and classification. With 400-pixel-by-240-pixel video input at 30 frames per second, the chip consumes 63 milliwatts.

A million spiking-neuron integrated circuit with a scalable communication network and interface

Heteroscedastic discriminant analysis and reduced rank HMMs for improved speech recognition

An overview of the current-mode approach for designing analog VLSI neural systems in subthreshold CMOS technology is presented. Emphasis is given to design techniques at the device level using the current-controlled current conveyor and the translinear principle. Circuits for associative memory and silicon retina systems are used as examples. The design methodology and how it relates to actual biological microcircuits are discussed. >

Current-mode subthreshold MOS circuits for analog VLSI neural systems

The goal of perception is to extract invariant properties of the underlying world. By computing contrast at edges, the retina reduces incident light intensities spanning twelve decades to a twentyfold variation. In one stroke, it solves the dynamic range problem and extracts relative reflectivity, bringing us a step closer to the goal. We have built a contrast-sensitive silicon retina that models all major synaptic interactions in the outer-plexiform layer of the vertebrate retina using current-mode CMOS circuits: namely, reciprocal synapses between cones and horizontal cells, which produce the antagonistic center/surround receptive field, and cone and horizontal cell gap junctions, which determine its size. The chip has 90 × 92 pixels on a 6.8 × 6.9mm die in 2µm n-well technology and is fully functional.

/pdf/a-contrast-sensitive-silicon-retina-with-reciprocal-synapses-1xz8hmrut6.pdf

A Contrast Sensitive Silicon Retina with Reciprocal Synapses

Biologically motivated feature extraction algorithms have been found to provide significantly robust performance in speech recognition systems, in the presence of channel and noise degradation, when compared to the standard features such as mel-cepstrum coefficients. However, auditory feature extraction is computationally expensive, and makes these features useless for real-time speech recognition systems.
In this thesis, I investigate the use of low power techniques and custom analog VLSI for auditory feature extraction. I first investigated the basilar-membrane model and the hair-cell model chips that were designed by Liu (Liu, 1992). I performed speech recognition experiments to evaluate how well these chips would perform as a front-end to a speech recognizer. Based on the experience gained by these experiments, I propose an alternate architecture that goes beyond the basilar-membrane model, and, using which, auditory features can be computed in real time. These chips have been designed and tested, and consume only a few milliwatts of power as compared to general purpose digital machines that consume several Watts.
I have also investigated Linear Discriminant Analysis (LDA) for dimension reduction of auditory features. Researchers have used Fisher-Rao linear discriminant analysis (LDA) to reduce the feature dimension. They model the low-dimensional features obtained from LDA as the outputs of a Markov process with hidden states (HMM). I present a unified framework for the problem of dimension reduction and HMM parameter estimation by modeling the original features with reduced-rank HMM. This re-formulation also leads to a generalization of LDA that is consistent with the heteroscedastic state models used in HMM, and give better performance when tested on a digit recognition task.

Investigation of silicon auditory models and generalization of linear discriminant analysis for improved speech recognition

Polarization is a general descriptor of light and contains information about reflecting objects that traditional intensity-based sensors ignore. Difficult computer vision tasks such as image segmentation and object orientation are made tractable with polarization vision techniques. Specularities, occluding contours, and material properties can be readily extracted if the Stokes polarization parameters are available. Astrophysicists employ polarization information to measure the spatial distribution of magnetic fields on the surface of the Sun. In the medical field, analysis of the polarization allows the diagnose of disease in the eyes. The retinae of most insect and certain vertebrate species are sensitive to polarization in their environment, but humans are blind to this property of light. Biologists use polarimeters to investigate behaviors of animals-vis-a-vis polarization-in their natural habitats. In this paper, we first present the basics of polarization sensing and then discuss integrated polarization imaging sensors developed in our laboratory.

Andreas G. Andreou

Papers

Heteroscedastic discriminant analysis and reduced rank HMMs for improved speech recognition

Current-mode subthreshold MOS circuits for analog VLSI neural systems

A Contrast Sensitive Silicon Retina with Reciprocal Synapses

Investigation of silicon auditory models and generalization of linear discriminant analysis for improved speech recognition

Polarization imaging: principles and integrated polarimeters