Pretrained contextualized embeddings are powerful word representations for structured prediction tasks. Recent work found that better word representations can be obtained by concatenating different types of embeddings. However, the selection of embeddings to form the best concatenated representation usually varies depending on the task and the collection of candidate embeddings, and the ever-increasing number of embedding types makes it a more difficult problem. In this paper, we propose Automated Concatenation of Embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured prediction tasks, based on a formulation inspired by recent progress on neural architecture search. Specifically, a controller alternately samples a concatenation of embeddings, according to its current belief of the effectiveness of individual embedding types in consideration for a task, and updates the belief based on a reward. We follow strategies in reinforcement learning to optimize the parameters of the controller and compute the reward based on the accuracy of a task model, which is fed with the sampled concatenation as input and trained on a task dataset. Empirical results on 6 tasks and 21 datasets show that our approach outperforms strong baselines and achieves state-of-the-art performance with fine-tuned embeddings in all the evaluations.

Automated Concatenation of Embeddings for Structured Prediction

Aspect-based sentiment analysis is a fine-grained sentiment classification task. Recently, graph neural networks over dependency trees have been explored to explicitly model connections between aspects and opinion words. However, the improvement is limited due to the inaccuracy of the dependency parsing results and the informal expressions and complexity of online reviews. To overcome these challenges, in this paper, we propose a dual graph convolutional networks (DualGCN) model that considers the complementarity of syntax structures and semantic correlations simultaneously. Particularly, to alleviate dependency parsing errors, we design a SynGCN module with rich syntactic knowledge. To capture semantic correlations, we design a SemGCN module with self-attention mechanism. Furthermore, we propose orthogonal and differential regularizers to capture semantic correlations between words precisely by constraining attention scores in the SemGCN module. The orthogonal regularizer encourages the SemGCN to learn semantically correlated words with less overlap for each word. The differential regularizer encourages the SemGCN to learn semantic features that the SynGCN fails to capture. Experimental results on three public datasets show that our DualGCN model outperforms state-of-the-art methods and verify the effectiveness of our model.

/pdf/dual-graph-convolutional-networks-for-aspect-based-sentiment-2xky5d6qn9.pdf

Dual Graph Convolutional Networks for Aspect-based Sentiment Analysis

Neural text classification models typically treat output labels as categorical variables that lack description and semantics. This forces their parametrization to be dependent on the label set size...

GILE: A Generalized Input-Label Embedding for Text Classification

Lors de l'enquête de 2011, 3 questions ont été modifiées : la résidence antérieure des personnes, la période d'achèvement et la surface des logements. Les résultats statistiques du recensement, basés sur 5 enquêtes successives et qui nécessitent donc une continuité du questionnement, ont dû être adaptés à partir du millésime 2009 : les données correspondantes ont été supprimées (résidence antérieure) ou diffusées sous une forme agrégée (période d'achèvement et la surface). Depuis le millésime 2013, cette contrainte a disparu et les données sont à nouveau diffusées, selon les concepts et les modalités du nouveau questionnaire introduit en 2011 : la résidence antérieure est celle occupée le 1er janvier de l'année précédant l'enquête, et non plus 5 ans auparavant ; les enfants nés après cette date sont considérés hors champ. la période d'achèvement du logement comprend les 6 modalités suivantes : avant 1919 ; de 1919 à 1945 ; de 1946 à 1970 ; de 1971 à 1990 ; de 1991 à 2005 ; de 2006 à 2011. la surface du logement est répartie selon 7 tranches : moins de 30 m2 ; de 30 à moins de 40 m2 ; de 40 à moins de 60 m2 ; de 60 à moins de 80 m2 ; de 80 à moins de 100 m2 ; de 100 à moins de 120 m2 ; 120 m2 ou plus. Dans les trois cas, les résultats du RP2014 ne sont pas comparables à ceux des RP 2006 à 2008.

Recensement de la population

The deep inside-outside recursive autoencoder (DIORA; Drozdov et al. 2019) is a self-supervised neural model that learns to induce syntactic tree structures for input sentences *without access to labeled training data*. In this paper, we discover that while DIORA exhaustively encodes all possible binary trees of a sentence with a soft dynamic program, its vector averaging approach is locally greedy and cannot recover from errors when computing the highest scoring parse tree in bottom-up chart parsing. To fix this issue, we introduce S-DIORA, an improved variant of DIORA that encodes a single tree rather than a softly-weighted mixture of trees by employing a hard argmax operation and a beam at each cell in the chart. Our experiments show that through *fine-tuning* a pre-trained DIORA with our new algorithm, we improve the state of the art in *unsupervised* constituency parsing on the English WSJ Penn Treebank by 2.2-6% F1, depending on the data used for fine-tuning.

/pdf/unsupervised-parsing-with-s-diora-single-tree-encoding-for-2cn4jk9jcz.pdf

Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive Autoencoders

Attention mechanisms have improved the performance of NLP tasks while allowing models to remain explainable. Self-attention is currently widely used, however interpretability is difficult due to the numerous attention distributions. Recent work has shown that model representations can benefit from label-specific information, while facilitating interpretation of predictions. We introduce the Label Attention Layer: a new form of self-attention where attention heads represent labels. We test our novel layer by running constituency and dependency parsing experiments and show our new model obtains new state-of-the-art results for both tasks on both the Penn Treebank (PTB) and Chinese Treebank. Additionally, our model requires fewer self-attention layers compared to existing work. Finally, we find that the Label Attention heads learn relations between syntactic categories and show pathways to analyze errors.

/pdf/rethinking-self-attention-towards-interpretability-in-neural-1c9i2gmnl1.pdf

Rethinking Self-Attention: Towards Interpretability in Neural Parsing.

Attention mechanisms have improved the performance of NLP tasks while providing for appearance of model interpretability. Self-attention is currently widely used in NLP models, however it is difficult to interpret due to the numerous attention distributions. We hypothesize that model representations can benefit from label-specific information, while facilitating interpretation of predictions. We introduce the Label Attention Layer: a new form of self-attention where attention heads represent labels. We validate our hypothesis by running experiments in constituency and dependency parsing and show our new model obtains new state-of-the-art results for both tasks on the English Penn Treebank. Our neural parser obtains 96.34 F1 score for constituency parsing, and 97.33 UAS and 96.29 LAS for dependency parsing. Additionally, our model requires fewer layers, therefore, fewer parameters compared to existing work.

Rethinking Self-Attention: An Interpretable Self-Attentive Encoder-Decoder Parser.

In this paper, we present a robotic approach to improve the teaching of handwriting using the tangible, haptic-enabled and classroom-friendly Cellulo robots. Our efforts presented here are in line with the philosophy of the Cellulo platform: we aim to create a ready-to-use tool (i.e. a set of robot-assisted activities) to be used for teaching handwriting, one that is to coexist harmoniously with traditional tools and will contribute new added values to the learning process, complementing existing teaching practices.To maximize our potential contributions to this learning process, we focus on two promising aspects of handwriting: the visual perception and the visual-motor coordination. These two aspects enhance in particular two sides of the representation of letters in the mind of the learner: the shape of the letter (the grapheme) and the way it is drawn, namely the dynamics of the letter (the ductus).With these two aspects in mind, we do a detailed content analysis for the process of learning the representation of letters, which leads us to discriminate the specific skills involved in letter representation. We then compare our robotic method with traditional methods as well as with the combination of the two methods, in order to discover which of these skills can benefit from the use of Cellulo.As handwriting is taught from age 5, we conducted our experiments with 17 five-year-old children in a public school. Results show a clear potential of our robot-assisted learning activities, with a visible improvement in certain skills of handwriting, most notably in creating the ductus of the letters, discriminating a letter among others and in the average handwriting speed.Moreover, we show that the benefit of our learning activities to the handwriting process increases when it is used after traditional learning methods. These results lead to the initial insights into how such a tangible robotic learning technology may be used to create cost-effective collaborative scenarios for the learning of handwriting.

/pdf/bringing-letters-to-life-handwriting-with-haptic-enabled-2h6nnx8ymz.pdf

Bringing letters to life: handwriting with haptic-enabled tangible robots

Understanding the intent of medical questions asked by patients, or Consumer Health Questions, is an essential skill for medical Conversational AI systems. We propose a novel data-augmented and simple joint learning approach combining question summarization and Recognizing Question Entailment (RQE) in the medical domain. Our data augmentation approach enables to use just one dataset for joint learning. We show improvements on both tasks across four biomedical datasets in accuracy (+8%), ROUGE-1 (+2.5%) and human evaluation scores. Human evaluation shows joint learning generates faithful and informative summaries. Finally, we release our code, the two question summarization datasets extracted from a large-scale medical dialogue dataset, as well as our augmented datasets.

/pdf/joint-summarization-entailment-optimization-for-consumer-1ux7oxofrd.pdf

Khalil Mrini

Papers

Rethinking Self-Attention: Towards Interpretability in Neural Parsing.

Rethinking Self-Attention: An Interpretable Self-Attentive Encoder-Decoder Parser.

Bringing letters to life: handwriting with haptic-enabled tangible robots

Joint Summarization-Entailment Optimization for Consumer Health Question Understanding

Rethinking Self-Attention: Towards Interpretability in Neural Parsing.