Word sense disambiguation (WSD) is the ability to identify the meaning of words in context in a computational manner. WSD is considered an AI-complete problem, that is, a task whose solution is at least as hard as the most difficult problems in artificial intelligence. We introduce the reader to the motivations for solving the ambiguity of words and provide a description of the task. We overview supervised, unsupervised, and knowledge-based approaches. The assessment of WSD systems is discussed in the context of the Senseval/Semeval campaigns, aiming at the objective evaluation of systems participating in several different disambiguation tasks. Finally, applications, open problems, and future directions are discussed.

/pdf/word-sense-disambiguation-a-survey-20csyn4stq.pdf

Word sense disambiguation: A survey

Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.

Longformer: The Long-Document Transformer

This paper introduces the use of Wikipedia as a resource for automatic keyword extraction and word sense disambiguation, and shows how this online encyclopedia can be used to achieve state-of-the-art results on both these tasks. The paper also shows how the two methods can be combined into a system able to automatically enrich a text with links to encyclopedic knowledge. Given an input document, the system identifies the important concepts in the text and automatically links these concepts to the corresponding Wikipedia pages. Evaluations of the system show that the automatic annotations are reliable and hardly distinguishable from manual annotations.

/pdf/wikify-linking-documents-to-encyclopedic-knowledge-42npyy9iyj.pdf

Wikify!: linking documents to encyclopedic knowledge

This paper describes experiments concerned with the automatic analysis of emotions in text. We describe the construction of a large data set annotated for six basic emotions: ANGER, DISGUST, FEAR, JOY, SADNESS and SURPRISE, and we propose and evaluate several knowledge-based and corpusbased methods for the automatic identification of these emotions in text.

/pdf/learning-to-identify-emotions-in-text-11xp6ggouo.pdf

Learning to identify emotions in text

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

/pdf/a-primer-in-bertology-what-we-know-about-how-bert-works-2v8af1zw7o.pdf

A Primer in BERTology: What We Know About How BERT Works

This paper explores the role of domain information in word sense disambiguation. The underlying hypothesis is that domain labels, such as MEDICINE, ARCHITECTURE and SPORT, provide a useful way to establish semantic relations among word senses, which can be profitably used during the disambiguation process. Results obtained at the SENSEVAL-2 initiative confirm that for a significant subset of words domain information can be used to disambiguate with a very high level of precision.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S1351324902003029

The role of domain information in Word Sense Disambiguation

In this paper we present a supervised Word Sense Disambiguation methodology, that exploits kernel methods to model sense distinctions. In particular a combination of kernel functions is adopted to estimate independently both syntagmatic and domain similarity. We defined a kernel function, namely the Domain Kernel, that allowed us to plug "external knowledge" into the supervised learning process. External knowledge is acquired from unlabeled data in a totally unsupervised way, and it is represented by means of Domain Models. We evaluated our methodology on several lexical sample tasks in different languages, outperforming significantly the state-of-the-art for each of them, while reducing the amount of labeled training data required for learning.

/pdf/domain-kernels-for-word-sense-disambiguation-ghkygg72y1.pdf

Domain Kernels for Word Sense Disambiguation

This is an electronic version of the paper presented at the International Florida Artificial Intelligence Research Society Conference, FLAIRS 2005

/pdf/automatic-assessment-of-students-free-text-answers-4i6lumu4yc.pdf

Automatic assessment of students’ free-text answers underpinned by the combination of a BLEU-inspired algorithm and latent semantic analysis

The major goal in ITC-irst's participation at Senseval-2 was to test the role of domain information in word sense disambiguation. The underlying working hypothesis is that domain labels, such as Medicine, Architecture and Sport provide a natural way to establish semantic relations among word senses, which can be profitably used during the disambiguation process. For each task in which we participated (i.e. English all words, English 'lexical sample' and Italian 'lexical sample') a different mix of knowledge based and statistical techniques were implemented.

Using Domain Information for Word Sense Disambiguation

This paper summarizes IRST’s participation in Senseval-3. We participated both in the English allwords task and in some lexical sample tasks (English, Basque, Catalan, Italian, Spanish). We followed two perspectives. On one hand, for the allwords task, we tried to refine the Domain Driven Disambiguation that we presented at Senseval-2. The refinements consist of both exploiting a new technique (Domain Relevance Estimation) for domain detection in texts, and experimenting with the use of Latent Semantic Analysis to avoid reliance on manually annotated domain resources (e.g. WORDNET DOMAINS). On the other hand, for the lexical sample tasks, we explored the direction of pattern abstraction and we demonstrated the feasibility of leveraging external knowledge using kernel methods.

Alfio Gliozzo

Papers

The role of domain information in Word Sense Disambiguation

Domain Kernels for Word Sense Disambiguation

Automatic assessment of students’ free-text answers underpinned by the combination of a BLEU-inspired algorithm and latent semantic analysis

Using Domain Information for Word Sense Disambiguation

Pattern abstraction and term similarity for Word Sense Disambiguation: IRST at Senseval-3