Attention Model has now become an important concept in neural networks that has been researched within diverse application domains. This survey provides a structured and comprehensive overview of the developments in modeling attention. In particular, we propose a taxonomy which groups existing techniques into coherent categories. We review salient neural architectures in which attention has been incorporated, and discuss applications in which modeling attention has shown a significant impact. We also describe how attention has been used to improve the interpretability of neural networks. Finally, we discuss some future research directions in attention. We hope this survey will provide a succinct introduction to attention models and guide practitioners while developing approaches for their applications.

An Attentive Survey of Attention Models

Prevailing methods for mapping large generative language models to supervised tasks may fail to sufficiently probe models’ novel capabilities. Using GPT-3 as a case study, we show that 0-shot prompts can significantly outperform few-shot prompts. We suggest that the function of few-shot examples in these cases is better described as locating an already learned task rather than meta-learning. This analysis motivates rethinking the role of prompts in controlling and evaluating powerful language models. We discuss methods of prompt programming, emphasizing the usefulness of considering prompts through the lens of natural language. We explore techniques for exploiting the capacity of narratives and cultural anchors to encode nuanced intentions and techniques for encouraging deconstruction of a problem into components before producing a verdict. Informed by this more encompassing theory of prompt programming, we also introduce the idea of a metaprompt that seeds the model to generate its own natural language prompts for a range of tasks. Finally, we discuss how these more general methods of interacting with language models can be incorporated into existing and future benchmarks and practical applications.

Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm

Answering complex questions is a time-consuming activity for humans that requires reasoning and integration of information. Recent work on reading comprehension made headway in answering simple questions, but tackling complex questions is still an ongoing research challenge. Conversely, semantic parsers have been successful at handling compositionality, but only when the information resides in a target knowledge-base. In this paper, we present a novel framework for answering broad and complex questions, assuming answering simple questions is possible using a search engine and a reading comprehension model. We propose to decompose complex questions into a sequence of simple questions, and compute the final answer from the sequence of answers. To illustrate the viability of our approach, we create a new dataset of complex questions, ComplexWebQuestions, and present a model that decomposes questions and interacts with the web to compute an answer. We empirically demonstrate that question decomposition improves performance from 20.8 precision@1 to 27.5 precision@1 on this new dataset.

The Web as a Knowledge-base for Answering Complex Questions

Aspect-based sentiment analysis is a fine-grained sentiment analysis task, which needs to detection the sentiment polarity towards a given aspect. Recently, graph neural models over the dependency tree are widely applied for aspect-based sentiment analysis. Most existing works, however, they generally focus on learning the dependency information from contextual words to aspect words based on the dependency tree of the sentence, which lacks the exploitation of contextual affective knowledge with regard to the specific aspect. In this paper, we propose a graph convolutional network based on SenticNet to leverage the affective dependencies of the sentence according to the specific aspect, called Sentic GCN . To be specific, we explore a novel solution to construct the graph neural networks via integrating the affective knowledge from SenticNet to enhance the dependency graphs of sentences. Based on it, both the dependencies of contextual words and aspect words and the affective information between opinion words and the aspect are considered by the novel affective enhanced graph model. Experimental results on multiple public benchmark datasets illustrate that our proposed model can beat state-of-the-art methods. 

Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks

Aspect-based sentiment analysis is a fine-grained sentiment analysis task, which needs to detection the sentiment polarity towards a given aspect. Recently, graph neural models over the dependency tree are widely applied for aspect-based sentiment analysis. Most existing works, however, they generally focus on learning the dependency information from contextual words to aspect words based on the dependency tree of the sentence, which lacks the exploitation of contextual affective knowledge with regard to the specific aspect. In this paper, we propose a graph convolutional network based on SenticNet to leverage the affective dependencies of the sentence according to the specific aspect, called Sentic GCN. To be specific, we explore a novel solution to construct the graph neural networks via integrating the affective knowledge from SenticNet to enhance the dependency graphs of sentences. Based on it, both the dependencies of contextual words and aspect words and the affective information between opinion words and the aspect are considered by the novel affective enhanced graph model. Experimental results on multiple public benchmark datasets illustrate that our proposed model can beat state-of-the-art methods.

Aspect-based sentiment analysis aims to determine the sentiment polarity towards a specific aspect in online reviews. Most recent efforts adopt attention-based neural network models to implicitly connect aspects with opinion words. However, due to the complexity of language and the existence of multiple aspects in a single sentence, these models often confuse the connections. In this paper, we address this problem by means of effective encoding of syntax information. Firstly, we define a unified aspect-oriented dependency tree structure rooted at a target aspect by reshaping and pruning an ordinary dependency parse tree. Then, we propose a relational graph attention network (R-GAT) to encode the new tree structure for sentiment prediction. Extensive experiments are conducted on the SemEval 2014 and Twitter datasets, and the experimental results confirm that the connections between aspects and opinion words can be better established with our approach, and the performance of the graph attention network (GAT) is significantly improved as a consequence.

Relational Graph Attention Network for Aspect-based Sentiment Analysis

/pdf/relational-graph-attention-network-for-aspect-based-2ce4akt04o.pdf

The success of neural summarization models stems from the meticulous encodings of source articles. To overcome the impediments of limited and sometimes noisy training data, one promising direction is to make better use of the available training data by applying filters during summarization. In this paper, we propose a novel Bi-directional Selective Encoding with Template (BiSET) model, which leverages template discovered from training data to softly select key information from each source article to guide its summarization process. Extensive experiments on a standard summarization dataset are conducted and the results show that the template-equipped BiSET model manages to improve the summarization performance significantly with a new state of the art.

/pdf/biset-bi-directional-selective-encoding-with-template-for-1slo5kxnuz.pdf

BiSET: Bi-directional Selective Encoding with Template for Abstractive Summarization

Generating fluent and informative responses is of critical importance for task-oriented dialogue systems. Existing pipeline approaches generally predict multiple dialogue acts first and use them to assist response generation. There are at least two shortcomings with such approaches. First, the inherent structures of multi-domain dialogue acts are neglected. Second, the semantic associations between acts and responses are not taken into account for response generation. To address these issues, we propose a neural co-generation model that generates dialogue acts and responses concurrently. Unlike those pipeline approaches, our act generation module preserves the semantic structures of multi-domain dialogue acts and our response generation module dynamically attends to different acts as needed. We train the two modules jointly using an uncertainty loss to adjust their task weights adaptively. Extensive experiments are conducted on the large-scale MultiWOZ dataset and the results show that our model achieves very favorable improvement over several state-of-the-art models in both automatic and human evaluations.

Kai Wang

Papers

Relational Graph Attention Network for Aspect-based Sentiment Analysis

Relational Graph Attention Network for Aspect-based Sentiment Analysis

BiSET: Bi-directional Selective Encoding with Template for Abstractive Summarization

Multi-Domain Dialogue Acts and Response Co-Generation

Multi-Domain Dialogue Acts and Response Co-Generation