We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model.

/pdf/latent-dirichlet-allocation-4gmscb06en.pdf

Latent dirichlet allocation

Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic, but the origin of these regularities has remained opaque. We analyze and make explicit the model properties needed for such regularities to emerge in word vectors. The result is a new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods. Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word cooccurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus. The model produces a vector space with meaningful substructure, as evidenced by its performance of 75% on a recent word analogy task. It also outperforms related models on similarity tasks and named entity recognition.

/pdf/glove-global-vectors-for-word-representation-55l84hyy3b.pdf

Glove: Global Vectors for Word Representation

We propose a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams [6], and Hof-mann's aspect model, also known as probabilistic latent semantic indexing (pLSI) [3]. In the context of text modeling, our model posits that each document is generated as a mixture of topics, where the continuous-valued mixture proportions are distributed as a latent Dirichlet random variable. Inference and learning are carried out efficiently via variational algorithms. We present empirical results on applications of this model to problems in text modeling, collaborative filtering, and text classification.

/pdf/latent-dirichlet-allocation-rkwbqihm3p.pdf

Latent Dirichlet Allocation

The increasing volume of data in modern business and science calls for more complex and sophisticated tools. Although advances in data mining technology have made extensive data collection much easier, it's still always evolving and there is a constant need for new techniques and tools that can help us transform this data into useful information and knowledge. Since the previous edition's publication, great advances have been made in the field of data mining. Not only does the third of edition of Data Mining: Concepts and Techniques continue the tradition of equipping you with an understanding and application of the theory and practice of discovering patterns hidden in large data sets, it also focuses on new, important topics in the field: data warehouses and data cube technology, mining stream, mining social networks, and mining spatial, multimedia and other complex data. Each chapter is a stand-alone guide to a critical topic, presenting proven algorithms and sound implementations ready to be used directly or with strategic modification against live data. This is the resource you need if you want to apply today's most powerful data mining techniques to meet real business challenges. * Presents dozens of algorithms and implementation examples, all in pseudo-code and suitable for use in real-world, large-scale data mining projects. * Addresses advanced topics such as mining object-relational databases, spatial databases, multimedia databases, time-series databases, text databases, the World Wide Web, and applications in several fields. *Provides a comprehensive, practical look at the concepts and techniques you need to get the most out of real business data

/pdf/data-mining-concepts-and-techniques-4dtvdfkvmi.pdf

Data Mining: Concepts and Techniques

High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such "autoencoder" networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data.

/pdf/reducing-the-dimensionality-of-data-with-neural-networks-51w34pmjfl.pdf

Reducing the Dimensionality of Data with Neural Networks

When searching for information in menu-based retrieval systems, users are given a choice among a set of categories. Successful retrieval of information depends critically on the user’s understanding the system’s division of objects into categories and the system’s labels for these categories. This research compares several different ways of describing ill-defined categories of objects using combinations of names and examples. Examples provide a promising possibility, both as a means of flexibly naming new or difficult menu categories and as a methodological tool for studying certain categorization problems.

https://link.springer.com/content/pdf/10.3758%2FBF03202396.pdf

Describing categories of objects for menu retrieval systems

Where there are texts in more than one language, it would be desirable if users could give queries or examples in the language in which they are most competent and obtain relevant text passages in any language. We have developed and tested a prototype system that makes this possible. The system is based entirely on a statistical technique that requires no humanly constructed dictionary, thesaurus, or term bank. The language-independent representation of text has two steps. In the rst, done just once for a subject area, a sample collection of parallel texts|paragraph-by-paragraph translations in two or more languages|is analyzed by a mathematical technique called Singular Value Decomposition. Each word in the sample is assigned a vector value determined by the total pattern of usage of all the words in all the sample paragraphs. In the second step, a new document or query in any of the original languages is assigned a vector value that is an average of the values of the words it contains. Tests on a French-English corpus showed that the method works well. Key-words: Interlingua, IR, information retrieval, statistical techniques, LSI, SVD, semantics, translation, multilingual lters

A statistical method for language-independent representation of the topical content of text segments

Problemes psychologiques lies a l'acces aux systemes de bases de donnees, a partir d'etudes experimentales. On discute les caracteristiques de la memoire humaine, de l'utilisation du langage et de la resolution de problemes

Human factors in data access

Retention of material learned in college-style technical courses was measured by a repeated final examination given one year later. Matched groups of students also repeated the final examination six weeks or six months after the end of the courses. The six-month group showed almost no loss over the year. The six-week group suffered about the same considerable loss as did a control group with no intervening examination. Questionnaire measures indicated that students who used over 10 percent of the examined material in their average monthly work also forgot very little. Apparently, course-acquired knowledge can be kept fresh over long periods by small amounts of properly spaced review activities.

Exams and Use as Preservatives of Course-Acquired Knowledge

Taken together with earlier work on command naming, Grudin and Barnard's results suggest that “specificity” and “meaningfulness” may be important while “naturalness” is not. However, important methodological problems are raised by experiments like this, in which conditions are arranged to maximize the effect of theoretically interesting variables while reducing overall similarity to real tasks.

http://hfs.sagepub.com/content/26/4/423.full.pdf

Thomas K. Landauer

Papers

Describing categories of objects for menu retrieval systems

A statistical method for language-independent representation of the topical content of text segments

Human factors in data access

Exams and Use as Preservatives of Course-Acquired Knowledge

What Makes a Difference When? Comments on Grudin and Barnard