Data Mining Practical Machine Learning Tools and Techniques

Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models to learn such representations  ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram, words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.

/pdf/enriching-word-vectors-with-subword-information-22sqq6w40j.pdf

Enriching Word Vectors with Subword Information

Semantic word spaces have been very useful but cannot express the meaning of longer phrases in a principled way. Further progress towards understanding compositionality in tasks such as sentiment detection requires richer supervised training and evaluation resources and more powerful models of composition. To remedy this, we introduce a Sentiment Treebank. It includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality. To address them, we introduce the Recursive Neural Tensor Network. When trained on the new treebank, this model outperforms all previous methods on several metrics. It pushes the state of the art in single sentence positive/negative classification from 80% up to 85.4%. The accuracy of predicting fine-grained sentiment labels for all phrases reaches 80.7%, an improvement of 9.7% over bag of features baselines. Lastly, it is the only model that can accurately capture the effects of negation and its scope at various tree levels for both positive and negative phrases.

/pdf/recursive-deep-models-for-semantic-compositionality-over-a-35h7g7u52l.pdf

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character $n$-grams. A vector representation is associated to each character $n$-gram; words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows us to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.

We analyze skip-gram with negative-sampling (SGNS), a word embedding method introduced by Mikolov et al., and show that it is implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information (PMI) of the respective word and context pairs, shifted by a global constant. We find that another embedding method, NCE, is implicitly factorizing a similar matrix, where each cell is the (shifted) log conditional probability of a word given its context. We show that using a sparse Shifted Positive PMI word-context matrix to represent words improves results on two word similarity tasks and one of two analogy tasks. When dense low-dimensional vectors are preferred, exact factorization with SVD can achieve solutions that are at least as good as SGNS's solutions for word similarity tasks. On analogy questions SGNS remains superior to SVD. We conjecture that this stems from the weighted nature of SGNS's factorization.

/pdf/neural-word-embedding-as-implicit-matrix-factorization-ltwfy3k99t.pdf

Neural Word Embedding as Implicit Matrix Factorization

In its most simple and informal formulation the principle of compositionality expresses an uncontrovertible truth of natural language interpretation, i.e. that the meaning of a complex expression can be derived from the meanings of its parts and the way these parts are syntactically related. The fact that our linguistic system is compositional is nicely confirmed by the capacity of speakers to use a restricted amount of linguistic resources to produce and understand a potentially unlimited number of sentences: “this would be impossible were we not able to distinguish parts in the thoughts corresponding to the parts of a sentence” (Frege 1923). The problem is then how to turn the intuitive statement of compositionality above into a formal constraint on the human language computational system. The lexicon raises some important issues that call into question the compositional nature of natural language interpretation. As Janssen (1997: 420) claims, “compositionality requires that words in isolation have a meaning and that from these meanings, the meaning of a compound can be built.” In other terms, the principle of compositionality seems to presuppose that word meaning is strictly context free, that is to say that “a lexical item must make approximately the same semantic contribution to each expression in which it occurs” (Fodor and Pylyshyn 1988). Prima facie this requirement sharply contrasts with another uncontrovertible fact about natural language, that is to say that lexical meanings are highly context sensitive. Actually, word meaning undergoes continuous modification and modulation in new contexts, and speakers have the ability to “adapt” the meaning of lexical items to fit a specific context and communicative situation (Pustejovsky 1998). A possible way to reconcile meaning variation in context with the requirements imposed by the compositional interpretation of complex expression is to address the former in terms of the notion of homonymy, i.e. the conflation of two symbolic entities under the same form. The fact that in the two sentences The director opened the bank and The water

The Lexicon and the Boundaries of Compositionality

This paper analyzes the concept of opposition and describes a fully unsupervised method for its automatic discrimination from near-synonymy in Distributional Semantic Models (DSMs). The discriminating method is based on the hypothesis that, even though both near-synonyms and opposites are mostly distributionally similar, opposites are different from each other in at least one dimension of meaning, which can be assumed to be salient. Such hypothesis has been implemented in APAnt, a distributional measure that evaluates the extent of the intersection among the most relevant contexts of two words (where relevance is measured as mutual dependency), and its saliency (i.e. their average rank in the mutual dependency sorted list of contexts). The measure – previously introduced in some pilot studies – is presented here with two variants. Evaluation shows that it outperforms three baselines in an antonym retrieval task: the vector cosine, a baseline implementing the co-occurrence hypothesis, and a random rank. This paper describes the algorithm in details and analyzes its current limitations, suggesting that extensions may be developed for discriminating antonyms not only from near-synonyms but also from other semantic relations. During the evaluation, we have noticed that APAnt also has a particular preference for hypernyms.

/pdf/when-similarity-becomes-opposition-synonyms-and-antonyms-4betone4o0.pdf

When Similarity Becomes Opposition: Synonyms and Antonyms Discrimination in DSMs

Argument alternations in Italian verbs: A computational study

This paper presents the design and results of a crowdsourcing experiment on the recognition of Italian event nominals. The aim of the experiment was to assess the feasibility of crowdsourcing methods for a complex semantic task such as distinguishing the eventive interpretation of polysemous nominals taking into consideration various types of syntagmatic cues. Details on the theoretical background and on the experiment set up are provided together with the final results in terms of accuracy and inter-annotator agreement. These results are compared with the ones obtained by expert annotators on the same task. The low values in accuracy and Fleiss kappa of the crowdsourcing experiment demonstrate that crowdsourcing is not always optimal for complex linguistic tasks. On the other hand, the use of non-expert contributors allows to understand what are the most ambiguous patterns of polysemy and the most useful syntagmatic cues to be used to identify the eventive reading of nominals.

/pdf/crowdsourcing-for-the-identification-of-event-nominals-an-4xlxllclo3.pdf

Crowdsourcing for the identification of event nominals: an experiment

English. In this paper, we illustrate the creation of a NER for the Public Administration (PA) domain. We discuss the creation of an annotated corpus with documents from the Italian Albo Pretorio Nazionale and provide results of the system evaluation. Italiano. In questo lavoro mostriamo la creazione di un NER per il dominio della Pubblica Amministrazione (PA). Presentiamo la creazione del corpus formato da documenti dell’Albo Pretorio Nazionale e mostriamo i risultati della valutazione del

/pdf/informed-pa-a-ner-for-the-italian-public-administration-415yvvw047.pdf

Alessandro Lenci

Papers

The Lexicon and the Boundaries of Compositionality

When Similarity Becomes Opposition: Synonyms and Antonyms Discrimination in DSMs

Argument alternations in Italian verbs: A computational study

Crowdsourcing for the identification of event nominals: an experiment

INFORMed PA: A NER for the Italian Public Administration Domain