Factorization approaches provide high accuracy in several important prediction problems, for example, recommender systems. However, applying factorization approaches to a new prediction problem is a nontrivial task and requires a lot of expert knowledge. Typically, a new model is developed, a learning algorithm is derived, and the approach has to be implemented.Factorization machines (FM) are a generic approach since they can mimic most factorization models just by feature engineering. This way, factorization machines combine the generality of feature engineering with the superiority of factorization models in estimating interactions between categorical variables of large domain. libFM is a software implementation for factorization machines that features stochastic gradient descent (SGD) and alternating least-squares (ALS) optimization, as well as Bayesian inference using Markov Chain Monto Carlo (MCMC). This article summarizes the recent research on factorization machines both in terms of modeling and learning, provides extensions for the ALS and MCMC algorithms, and describes the software tool libFM.

Factorization Machines with libFM

Over the past few years, neural networks have re-emerged as powerful machine-learning models, yielding state-of-the-art results in fields such as image recognition and speech processing. More recently, neural network models started to be applied also to textual natural language signals, again with very promising results. This tutorial surveys neural network models from the perspective of natural language processing research, in an attempt to bring natural-language researchers up to speed with the neural techniques. The tutorial covers input encoding for natural language tasks, feed-forward networks, convolutional networks, recurrent networks and recursive networks, as well as the computation graph abstraction for automatic gradient computation.

/pdf/a-primer-on-neural-network-models-for-natural-language-4uqm8o02v5.pdf

A primer on neural network models for natural language processing

Tagging plays an important role in many recent websites. Recommender systems can help to suggest a user the tags he might want to use for tagging a specific item. Factorization models based on the Tucker Decomposition (TD) model have been shown to provide high quality tag recommendations outperforming other approaches like PageRank, FolkRank, collaborative filtering, etc. The problem with TD models is the cubic core tensor resulting in a cubic runtime in the factorization dimension for prediction and learning.In this paper, we present the factorization model PITF (Pairwise Interaction Tensor Factorization) which is a special case of the TD model with linear runtime both for learning and prediction. PITF explicitly models the pairwise interactions between users, items and tags. The model is learned with an adaption of the Bayesian personalized ranking (BPR) criterion which originally has been introduced for item recommendation. Empirically, we show on real world datasets that this model outperforms TD largely in runtime and even can achieve better prediction quality. Besides our lab experiments, PITF has also won the ECML/PKDD Discovery Challenge 2009 for graph-based tag recommendation.

/pdf/pairwise-interaction-tensor-factorization-for-personalized-53romkykbu.pdf

Pairwise interaction tensor factorization for personalized tag recommendation

We present a simple and effective scheme for dependency parsing which is based on bidirectional-LSTMs (BiLSTMs). Each sentence token is associated with a BiLSTM vector representing the token in its sentential context, and feature vectors are constructed by concatenating a few BiLSTM vectors. The BiLSTM is trained jointly with the parser objective, resulting in very effective feature extractors for parsing. We demonstrate the effectiveness of the approach by applying it to a greedy transition-based parser as well as to a globally optimized graph-based parser. The resulting parsers have very simple architectures, and match or surpass the state-of-the-art accuracies on English and Chinese.

/pdf/simple-and-accurate-dependency-parsing-using-bidirectional-ksfm27bwl1.pdf

Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations

This work was sponsored in part by the U. S. Army Research Laboratory and the U. S. Army Research Office/nunder contract/grant number W911NF-10-1-0533, and in part by NSF CAREER grant IIS-1054319./nMiguel Ballesteros is supported by the European Commission under the contract numbers FP7-ICT-610411 (project MULTISENSOR) and H2020-RIA-645012 (project KRISTINA).

/pdf/transition-based-dependency-parsing-with-stack-long-short-24hvn48dnj.pdf

Transition-Based Dependency Parsing with Stack Long Short-Term Memory

In the absence of annotations in the target language, multilingual models typically draw on extensive parallel resources. In this paper, we demonstrate that accurate multilingual partof-speech (POS) tagging can be done with just a few (e.g., ten) word translation pairs. We use the translation pairs to establish a coarse linear isometric (orthonormal) mapping between monolingual embeddings. This enables the supervised source model expressed in terms of embeddings to be used directly on the target language. We further refine the model in an unsupervised manner by initializing and regularizing it to be close to the direct transfer model. Averaged across six languages, our model yields a 37.5% absolute improvement over the monolingual prototypedriven method (Haghighi and Klein, 2006) when using a comparable amount of supervision. Moreover, to highlight key linguistic characteristics of the generated tags, we use them to predict typological properties of languages, obtaining a 50% error reduction relative to the prototype model.1

/pdf/ten-pairs-to-tag-multilingual-pos-tagging-via-coarse-mapping-1cwav6emlb.pdf

Ten pairs to tag - Multilingual POS tagging via coarse mapping between embeddings

Accurate scoring of syntactic structures such as head-modifier arcs in dependency parsing typically requires rich, highdimensional feature representations. A small subset of such features is often selected manually. This is problematic when features lack clear linguistic meaning as in embeddings or when the information is blended across features. In this paper, we use tensors to map high-dimensional feature vectors into low dimensional representations. We explicitly maintain the parameters as a low-rank tensor to obtain low dimensional representations of words in their syntactic roles, and to leverage modularity in the tensor for easy training with online algorithms. Our parser consistently outperforms the Turbo and MST parsers across 14 different languages. We also obtain the best published UAS results on 5 languages. 1

/pdf/low-rank-tensors-for-scoring-dependency-structures-2wb4engixn.pdf

Low-Rank Tensors for Scoring Dependency Structures

We introduce a neural method for transfer learning between two (source and target) classification tasks or aspects over the same domain. Rather than training on target labels, we use a few keywords pertaining to source and target aspects indicating sentence relevance instead of document class labels. Documents are encoded by learning to embed and softly select relevant sentences in an aspect-dependent manner. A shared classifier is trained on the source encoded documents and labels, and applied to target encoded documents. We ensure transfer through aspect-adversarial training so that encoded documents are, as sets, aspect-invariant.  Experimental results demonstrate that our approach outperforms different baselines and model variants on two datasets, yielding an improvement of 27% on a pathology dataset and 5% on a review dataset.

/pdf/aspect-augmented-adversarial-networks-for-domain-adaptation-596dhn4cfr.pdf

Aspect-augmented Adversarial Networks for Domain Adaptation

Marketing strategies without emotion will not work. Emotion stimulates the mind 3,000 times quicker than rational thought. Such emotion invokes either a positive or a negative response and physical expressions. Understanding the underlying dynamics of users' emotions can efficiently help companies formulate marketing strategies and support after-sale services. While prior work has focused mainly on qualitative aspects, in this paper we present our research on quantitative analysis of how an individual's emotional state can be inferred from her historic emotion log and how this person's emotional state influences (or is influenced by) her friends in the social network. We statistically study the dynamics of individual's emotions and discover several interesting as well as important patterns. Based on this discovery, we propose an approach referred to as MoodCast to learn to infer individuals' emotional states. In both mobile-based social network and online virtual network, we verify the effectiveness of our proposed approach.

http://keg.cs.tsinghua.edu.cn/jietang/publications/TAC11-Tang-et-al-social-emotion-analysis.pdf

Quantitative Study of Individual Emotional States in Social Networks

Traditional syntax models typically leverage part-of-speech (POS) information by constructing features from hand-tuned templates. We demonstrate that a better approach is to utilize POS tags as a regularizer of learned representations. We propose a simple method for learning a stacked pipeline of models which we call “stack-propagation”. We apply this to dependency parsing and tagging, where we use the hidden layer of the tagger network as a representation of the input tokens for the parser. At test time, our parser does not require predicted POS tags. On 19 languages from the Universal Dependencies, our method is 1.3% (absolute) more accurate than a state-of-the-art graph-based approach and 2.7% more accurate than the most comparable greedy model.

/pdf/stack-propagation-improved-representation-learning-for-34qpsculmg.pdf

Yuan Zhang

Papers

Ten pairs to tag - Multilingual POS tagging via coarse mapping between embeddings

Low-Rank Tensors for Scoring Dependency Structures

Aspect-augmented Adversarial Networks for Domain Adaptation

Quantitative Study of Individual Emotional States in Social Networks

Stack-propagation: Improved Representation Learning for Syntax