Word embedding

About: Word embedding is a research topic. Over the lifetime, 4683 publications have been published within this topic receiving 153378 citations. The topic is also known as: word embeddings.

Abstractive Arabic Text Summarization Based on Deep Learning

Abstract: Text summarization (TS) is considered one of the most difficult tasks in natural language processing (NLP). It is one of the most important challenges that stand against the modern computer system's capabilities with all its new improvement. Many papers and research studies address this task in literature but are being carried out in extractive summarization, and few of them are being carried out in abstractive summarization, especially in the Arabic language due to its complexity. In this paper, an abstractive Arabic text summarization system is proposed, based on a sequence-to-sequence model. This model works through two components, encoder and decoder. Our aim is to develop the sequence-to-sequence model using several deep artificial neural networks to investigate which of them achieves the best performance. Different layers of Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM), and Bidirectional Long Short-Term Memory (BiLSTM) have been used to develop the encoder and the decoder. In addition, the global attention mechanism has been used because it provides better results than the local attention mechanism. Furthermore, AraBERT preprocess has been applied in the data preprocessing stage that helps the model to understand the Arabic words and achieves state-of-the-art results. Moreover, a comparison between the skip-gram and the continuous bag of words (CBOW) word2Vec word embedding models has been made. We have built these models using the Keras library and run-on Google Colab Jupiter notebook to run seamlessly. Finally, the proposed system is evaluated through ROUGE-1, ROUGE-2, ROUGE-L, and BLEU evaluation metrics. The experimental results show that three layers of BiLSTM hidden states at the encoder achieve the best performance. In addition, our proposed system outperforms the other latest research studies. Also, the results show that abstractive summarization models that use the skip-gram word2Vec model outperform the models that use the CBOW word2Vec model.

Generative Text Summary Based on Enhanced Semantic Attention and Gain-Benefit Gate

Abstract: Generative text summary is an important branch of natural language processing. Aiming at the problems of insufficient use of semantic information, insufficient summary precision and the problem of semantics-loss in the current generated text summary method, an enhanced semantic model is proposed based on dual-encoder, which can provide richer semantic information for sequence-to-sequence architecture through dual-encoder. The enhanced attention architecture with dual-channel semantics is optimized, and the empirical distribution and Gain-Benefit gate are built for decoding. In addition, the position embedding and word embedding are merged into the word embedding technology, and the TF-IDF(term frequency-inverse document frequency), part of speech, key score are added to word’s feature. Meanwhile, the optimal dimension of word embedding is optimized. This paper aims to optimize the traditional sequence mapping and word feature representation, enhance the model’s semantic understanding, and improve the quality of the summary. The LCSTS and SOGOU datasets are used to validate proposed method. The experimental results show that the proposed method can improve the performance of the ROUGE evaluation system by 10–13 percentage points compared with other listed algorithms. We can observe that the semantic understanding of the text summaries is more accurate and the generation effect is better, which has a better application prospect.

Specializing Word Embeddings (for Parsing) by Information Bottleneck

Abstract: Pre-trained word embeddings like ELMo and BERT contain rich syntactic and semantic information, resulting in state-of-the-art performance on various tasks. We propose a very fast variational information bottleneck (VIB) method to nonlinearly compress these embeddings, keeping only the information that helps a discriminative parser. We compress each word embedding to either a discrete tag or a continuous vector. In the discrete version, our automatically compressed tags form an alternative tag set: we show experimentally that our tags capture most of the information in traditional POS tag annotations, but our tag sequences can be parsed more accurately at the same level of tag granularity. In the continuous version, we show experimentally that moderately compressing the word embeddings by our method yields a more accurate parser in 8 of 9 languages, unlike simple dimensionality reduction.

Hate Speech Detection using Fusion Approach

Abstract: Detection of hate speech in user-generated online content has become an issue of increasing importance in recent years and is discerning for applications such as disputed event identification and sentiment analysis. Text classification for online content is a bit challenging task due to the natural language complexity and hastily generated online user microblogs including a plethora of informality and mistakes. This work introduces a system to classify tweets in three categories (i.e., racism, sexism and none). In our classification strategy, we integrate deep features extracted from Convolutional Neural Network(CNN) trained on semantic word embedding with state-of-the-art syntactic and word n-gram features. We perform comprehensive experiments on a standard dataset containing 16k manually annotated tweets. Our proposed approach outperform all other state-of-the-art approaches with a significant increase in accuracy.

Focal elements of neural information retrieval models. An outlook through a reproducibility study

Abstract: This paper analyzes two state-of-the-art Neural Information Retrieval (NeuIR) models: the Deep Relevance Matching Model (DRMM) and the Neural Vector Space Model (NVSM). Our contributions include: (i) a reproducibility study of two state-of-the-art supervised and unsupervised NeuIR models, where we present the issues we encountered during their reproducibility; (ii) a performance comparison with other lexical, semantic and state-of-the-art models, showing that traditional lexical models are still highly competitive with DRMM and NVSM; (iii) an application of DRMM and NVSM on collections from heterogeneous search domains and in different languages, which helped us to analyze the cases where DRMM and NVSM can be recommended; (iv) an evaluation of the impact of varying word embedding models on DRMM, showing how relevance-based representations generally outperform semantic-based ones; (v) a topic-by-topic evaluation of the selected NeuIR approaches, comparing their performance to the well-known BM25 lexical model, where we perform an in-depth analysis of the different cases where DRMM and NVSM outperform the BM25 model or fail to do so. We run an extensive experimental evaluation to check if the improvements of NeuIR models, if any, over the selected baselines are statistically significant.