Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. These methods have dramatically improved the state-of-the-art in speech recognition, visual object recognition, object detection and many other domains such as drug discovery and genomics. Deep learning discovers intricate structure in large data sets by using the backpropagation algorithm to indicate how a machine should change its internal parameters that are used to compute the representation in each layer from the representation in the previous layer. Deep convolutional nets have brought about breakthroughs in processing images, video, speech and audio, whereas recurrent nets have shone light on sequential data such as text and speech.

Deep learning

Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

Setting of the learning problem consistency of learning processes bounds on the rate of convergence of learning processes controlling the generalization ability of learning processes constructing learning algorithms what is important in learning theory?.

The Nature of Statistical Learning Theory

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to ½ everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

/pdf/generative-adversarial-nets-1ofhan3sbf.pdf

Generative Adversarial Nets

In this paper we investigate the utility of three aspects of named entity processing: detection, localization and value extraction. We corroborate this task categorization by providing examples of practical applications for each of these subtasks. We also suggest methods for tackling these subtasks, giving particular attention to working with speech data. We employ Support Vector Machines to solve the detection task and show how localization and value extraction can successfully be dealt with using a combination of grammar-based and statistical methods.

/pdf/aspects-of-named-entity-processing-4rl2bqix2p.pdf

Aspects of named entity processing.

In this paper, we study the Location-based Reporting Tool (LRT), a smartphone application for collecting large-scale feedback from mobile customers. Using one-year data collected from one of the largest cellular networks in the US, we compare LRT feedback to the traditional customer feedback channel -- customer care tickets. Our analysis shows that, due to the light-weight design, LRT encourages customers to report more problems from anywhere and at any time. In addition, we find LRT users access network services more intensively than other mobile users, and hence are more likely to experience and are more sensitive to network problems. All these render LRT feedback a valuable information source for early detection of emerging network problems.

/pdf/large-scale-app-based-reporting-of-customer-problems-in-3z6u1orohh.pdf

Large-scale app-based reporting of customer problems in cellular networks: potential and limitations

Operating, managing and securing networks require a thorough understanding of the demands placed on the network by the endpoints it interconnects, the characteristics of the traffic the endpoints generate, and the distribution of that traffic over the resources of the network infrastructure. A major differentiator in the types of resource required by traffic is the class of endpoint application that generates it. Service providers determine the application mix present in traffic via measurements, e.g., flow measurements furnished by routers. Previous work has shown that a fairly accurate determination of application type can be made from this data. However, protocol level information, such as TCP/UDP ports and other parts of the transport header, and also parts of the network header in some cases, may not be accessible due to the use of encryption or tunneling protocols by endpoints or gateways. Furthermore, the utility of ports as signifiers of application type has some limitations due to abuse and non-standard usage, amongst other reasons. These factors reduce the classification accuracy. In this paper, we propose a novel technique for inferring the distribution of application classes present in the aggregated traffic flows between endpoints, that exploits both the measured statistics of the traffic flows, and the spatial distribution of those flows across the network. Our method employs a two-step supervised model, where the bootstrapping step provides initial (inaccurate) inference on the traffic application classes, and the graph-based calibration step adjusts the initial inference through the collective spatial traffic distribution. In evaluations using real traffic flow measurements from a large ISP, we show how our method can accurately classify application types within aggregate traffic between endpoints, even without knowledge of ports and other traffic features. While the bootstrap estimate classifies the aggregates with 80% accuracy, incorporating spatial distributions through calibration increases the accuracy to 92%, i.e., roughly halving the number of errors.

/pdf/inferring-applications-at-the-network-layer-using-collective-1oiz3dx028.pdf

Inferring applications at the network layer using collective traffic statistics

Nous presentons une technique nouvelle de compression d'images appelee DjVu. Cette technique est specialement concue pour la compression de documents en couleurs numerises a haute resolution. Un fichier DjVu representant une page typique d'un magazine en couleurs, numerisee a 300 points par pouce (dpi), requiert entre 40 et 80 KB, ce qui est est 5 a 10 fois meilleur qu'un fichier JPEG offrant une lisibilite similaire. Le compresseur DjVu commence par classer chaque pixel de l'image numerisee comme pixel d'avant-plan (texte, dessins au trait) ou pixel d'arriere-plan (images, photos, texture du papier). Cette classification forme une image bitonale qui est compressee grâce a une technique qui tire parti des similitudes de forme entre les divers caracteres composant l'avant-plan. Les images d'avant-plan et d'arriere-plan sont ensuite compressees a l'aide d'un algorithme a base d'ondelettes a resolution reduite. Un algorithme de masquage minimise le nombre de bits utilises pour coder les pixels d'avant-plan ou d'arriere-plan qui ne sont pas visibles dans l'image finale. Un plug-in pour butineur permet de visualiser tres efficacement les images DjVu sur le Web.

Djvu: Un systeme de compression d'images pour la distribution reticulaire de documents numerises (Djvu: An image compression system for distributing scanned document on the internet)

Image-based digital documents are composed of multiple pages, each of which may be composed of multiple components such as the text, pictures, background, and annotations. We describe the image structure and software architecture that allows the DjVu system to load and render the required components on demand while minimizing the bandwidth requirements, and the memory requirements in the client. DjVu document files are merely a list of enriched URLs that point to individual files (or file elements) that contain image components. Image components include: text images, background images, shape dictionaries shared by multiple pages, OCRed text, and several types of annotations. A multithreaded software architecture with smart caching allows individual components to be loaded and predecoded and rendered on-demand. Pages are pre-fetched or loaded on demand, allowing users to randomly access pages without downloading the entire document, and without the help of a byte server. Components that are shared accross pages (e.g. shape dictionnaries, or background layers) are loaded as required and cached This greatly reduces the overall bandwidth requirements. Shared dictionnaries allow 40% typical file size reduction for scanned bitonal documents at 300dpi. Compression ratios on scanned US patents at 300dpi are 5.2 to 10.2 times higher than GroupIV with shared dictionnaries and 3.6 to 8.5 times higher than GroupIV without shared dictionnaries.

Patrick Haffner

Papers

Aspects of named entity processing.

Large-scale app-based reporting of customer problems in cellular networks: potential and limitations

Inferring applications at the network layer using collective traffic statistics

Djvu: Un systeme de compression d'images pour la distribution reticulaire de documents numerises (Djvu: An image compression system for distributing scanned document on the internet)

DJVU document browsing with on-demand loading and rendering of image components