We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model.

/pdf/latent-dirichlet-allocation-4gmscb06en.pdf

Latent dirichlet allocation

We propose a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams [6], and Hof-mann's aspect model, also known as probabilistic latent semantic indexing (pLSI) [3]. In the context of text modeling, our model posits that each document is generated as a mixture of topics, where the continuous-valued mixture proportions are distributed as a latent Dirichlet random variable. Inference and learning are carried out efficiently via variational algorithms. We present empirical results on applications of this model to problems in text modeling, collaborative filtering, and text classification.

/pdf/latent-dirichlet-allocation-rkwbqihm3p.pdf

Latent Dirichlet Allocation

David M. Blei
Departments of Statistics and Computer Science, Columbia University, New York, NY 10027, USAMichael I. Jordan
Computer Science Division and Department of Statistics, University of California, Berkeley, CA
94720, USAWe present stochastic variational inference algorithms for two Bayesian nonnegative matrix factorization (NMF) models. These algorithms allow for fast processing of massive datasets. In particular,
we derive stochastic algorithms for a Bayesian extension of the NMF algorithm of Lee and Seung (2001), and a matrix factorization model called correlated NMF, which is motivated by the
correlated topic model (Blei and Lafferty, 2007). We apply our algorithms to roughly 1.8 million
documents from the New York Times, comparing with online LDA (Hoffman et al., 2010b).

/pdf/bayesian-nonnegative-matrix-factorization-with-stochastic-3e7tx2hn09.pdf

Bayesian Nonnegative Matrix Factorization with Stochastic Variational Inference

Many natural systems, such as neurons firing in the brain or basketball teams traversing a court, give rise to time series data with complex, nonlinear dynamics We can gain insight into these systems by decomposing the data into segments that are each explained by simpler dynamic units Building on switching linear dynamical systems (SLDS), we present a new model class that not only discovers these dynamical units, but also explains how their switching behavior depends on observations or continuous latent states These "recurrent" switching linear dynamical systems provide further insight by discovering the conditions under which each unit is deployed, something that traditional SLDS models fail to do We leverage recent algorithmic advances in approximate inference to make Bayesian inference in these models easy, fast, and scalable

Recurrent switching linear dynamical systems

We present the discrete innite logistic normal distribution (DILN), a Bayesian nonparametric prior for mixed membership models. DILN generalizes the hierarchical Dirichlet process (HDP) to model correlation structure between the weights of the atoms at the group level. We derive a representation of DILN as a normalized collection of gamma-distributed random variables and study its statistical properties. We derive a variational inference algorithm for approximate posterior inference. We apply DILN to topic modeling of documents and study its empirical performance on four corpora, comparing performance with the HDP and the correlated topic model (CTM). To compute with large-scale data, we develop a stochastic variational inference algorithm for DILN and compare with similar algorithms for HDP and latent Dirichlet allocation (LDA) on a collection of 350; 000 articles from Nature.

/pdf/the-discrete-innite-logistic-normal-distribution-crbyem8upm.pdf

The Discrete Innite Logistic Normal Distribution

Computer method and apparatus for segmenting text streams is disclosed. Given is an input text stream formed of a series of words. A probability member provides working probabilities that a group of words is of a topic selected from a plurality of predetermined topics. The probability member accounts for relationships between words. A processing module receives the input text stream and using the probability member determines probability of certain words in the input text stream being of a same topic. As such, the processing module segments the input text stream into single topic groupings of words, where each grouping is of a respective single topic.

Computer method and apparatus for segmenting text streams

Invited Presentations.- Structural Inference of Hierarchies in Networks.- Heider vs Simmel: Emergent Features in Dynamic Structures.- Joint Group and Topic Discovery from Relations and Text.- Statistical Models for Networks: A Brief Review of Some Recent Research.- Other Presentations.- Combining Stochastic Block Models and Mixed Membership for Statistical Network Analysis.- Exploratory Study of a New Model for Evolving Networks.- A Latent Space Model for Rank Data.- A Simple Model for Complex Networks with Arbitrary Degree Distribution and Clustering.- Discrete Temporal Models of Social Networks.- Approximate Kalman Filters for Embedding Author-Word Co-occurrence Data over Time.- Discovering Functional Communities in Dynamical Networks.- Empirical Analysis of a Dynamic Social Network Built from PGP Keyrings.- Extended Abstracts.- A Brief Survey of Machine Learning Methods for Classification in Networked Data and an Application to Suspicion Scoring.- Age and Geographic Inferences of the LiveJournal Social Network.- Inferring Organizational Titles in Online Communication.- Learning Approximate MRFs from Large Transactional Data.- Panel Discussion.- Panel Discussion.

David M. Blei

Papers

Bayesian Nonnegative Matrix Factorization with Stochastic Variational Inference

Recurrent switching linear dynamical systems

The Discrete Innite Logistic Normal Distribution

Computer method and apparatus for segmenting text streams

Statistical Network Analysis: Models, Issues, and New Directions