Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. These methods have dramatically improved the state-of-the-art in speech recognition, visual object recognition, object detection and many other domains such as drug discovery and genomics. Deep learning discovers intricate structure in large data sets by using the backpropagation algorithm to indicate how a machine should change its internal parameters that are used to compute the representation in each layer from the representation in the previous layer. Deep convolutional nets have brought about breakthroughs in processing images, video, speech and audio, whereas recurrent nets have shone light on sequential data such as text and speech.

Deep learning

Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

Setting of the learning problem consistency of learning processes bounds on the rate of convergence of learning processes controlling the generalization ability of learning processes constructing learning algorithms what is important in learning theory?.

The Nature of Statistical Learning Theory

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to ½ everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

/pdf/generative-adversarial-nets-1ofhan3sbf.pdf

Generative Adversarial Nets

Kernel methods have found in recent years wide use in statistical learning techniques due to their good performance and their computational efficiency in high-dimensional feature space. However, text or speech data cannot always be represented by the fixed-length vectors that the traditional kernels handle. We recently introduced a general kernel framework based on weighted transducers, rational kernels ,t o extend kernel methods to the analysis of variable-length sequences and weighted automata [5] and described their application to spoken-dialog applications. We presented a constructive algorithm for ensuring that rational kernels are positive definite symmetric, a property which guarantees the convergence of discriminant classification algorithms such as Support Vector Machines, and showed that many string kernels previously introduced in the computational biology literature are special instances of such positive definite symmetric rational kernels [4]. This paper reviews the essential results given in [5, 3, 4] and presents them in the form of a short tutorial.

Weighted automata kernels - General framework and algorithms

Disclosed herein are systems, methods, and computer-readable storage devices for building classifiers in a semi-supervised or unsupervised way. An example system implementing the method can receive a human-generated map which identifies categories of transcriptions. Then the system can receive a set of machine transcriptions. The system can process each machine transcription in the set of machine transcriptions via a set of natural language understanding classifiers, to yield a machine map, the machine map including a set of classifications and a classification score for each machine transcription in the set of machine transcriptions. Then the system can generate silver annotated data by combining the human-generated map and the machine map. The algorithm can include different branches for when the machine transcription is available, when partial results are available, when no results are found for the machine transcription, and so forth.

System and method for training a classifier for natural language understanding

Using distinct and appropriate synthetic voices to voice the characters in a children’s story would make a TTS-based digital storyteller system more engaging and entertaining, as well increase listener’s comprehension of the story. However, automatically predicting appropriate voices for storybook characters is both a non-trivial and largely unexplored problem. In this paper, we present a data-driven approach for predicting the most appropriate voices for characters in children’s stories based on salient character attributes. We use Mechanical Turk to identify the character attributes that are most salient in evoking the listeners’ perception that a specific character should have a particular voice, and to label the voices in our collection with attribute tags. We model the attribute-to-voice relationship with Naive Bayes. The resulting system performs significantly above chance in an objective evaluation, demonstrating the viability of our approach.

Predicting Character-Appropriate Voices for a TTS-based Storyteller System.

Disclosed herein are systems, methods, and non-transitory computer-readable storage media for combining frame and segment level processing, via temporal pooling, for phonetic classification A frame processor unit receives an input and extracts the time-dependent features from the input A plurality of pooling interface units generates a plurality of feature vectors based on pooling the time-dependent features and selecting a plurality of time-dependent features according to a plurality of selection strategies Next, a plurality of segmental classification units generates scores for the feature vectors Each segmental classification unit (SCU) can be dedicated to a specific pooling interface unit (PIU) to form a PIU-SCU combination Multiple PIU-SCU combinations can be further combined to form an ensemble of combinations, and the ensemble can be diversified by varying the pooling operations used by the PIU-SCU combinations Based on the scores, the plurality of segmental classification units selects a class label and returns a result

System and method for combining frame and segment level processing, via temporal pooling, for phonetic classification

In one embodiment, the present disclosure is a method and apparatus for classifying applications using the collective properties of network traffic. In one embodiment, a method for classifying traffic in a communication network includes receiving a traffic activity graph, the traffic activity graph comprising a plurality of nodes interconnected by a plurality of edges, where each of the nodes represents an endpoint associated with the communication network and each of the edges represents traffic between a corresponding pair of the nodes, generating an initial set of inferences as to an application class associated with each of the edges, based on at least one measured statistic related to at least one traffic flow in the communication network, and refining the initial set of inferences based on a spatial distribution of the traffic flows, to produce a final traffic activity graph.

Patrick Haffner

Papers

Weighted automata kernels - General framework and algorithms

System and method for training a classifier for natural language understanding

Predicting Character-Appropriate Voices for a TTS-based Storyteller System.

System and method for combining frame and segment level processing, via temporal pooling, for phonetic classification

Method and apparatus for classifying applications using the collective properties of network traffic