scispace - formally typeset
Search or ask a question
Journal ArticleDOI

A Sentiment Information Collector-Extractor Architecture Based Neural Network for Sentiment Analysis

TL;DR: A new ensemble strategy is applied to combine the results of different sub-extractors, making the SIE more universal and outperform any single sub- Extractor and outperforms the state-of-the-art methods on three datasets of different language.
About: This article is published in Information Sciences.The article was published on 2018-10-01 and is currently open access. It has received 21 citations till now. The article focuses on the topics: Sentiment analysis & Deep learning.

Summary (3 min read)

1. Introduction

  • Deep learning has made a great progress recently and plays an important role in academia and industry.
  • This key word appears in two completely different positions.
  • Besides, sentence (iii) contains two key words not and pleasant and they are separated by another word been.
  • How to locate the key words remains a big challenge in sentiment analysis.30 Researchers have designed many efficient models in order to capture the sentiment information.
  • Thus, it could reduce the effectiveness40 when RNN is used to capture the semantics of a whole sentence, because key components could appear anywhere in a sentence rather than at the end.

3. Model

  • Figure 1 shows the architecture of the whole model.
  • As is illustrated in Figure 1, the model can be divided into two part: (i) SIC and (ii) SIE.
  • Then the matrix X is fed into information extractor and latent semantic information will be extracted based on model ensemble strategy.

3.1. Sentiment Information Collector (SIC)

  • The authors first describe the architecture of the SIC in their model.
  • The left-side context cl(vi) of word vi is calculated using Equation(1), where e(vi) is the word embedding of word vi, which is a dense vector with |e| real value elements.
  • The information extractor, which is an ensemble model, is designed to extract sentiment information precisely from sentence information matrix X. The SIE consists of three subextractors.
  • In their case, the authors choose ReLU [31] as the nonlinear function.
  • When all of the latent semantic vectors mji are calculated separately, each sub-extractor will apply a max-pooling operation: mj = L max i=1 mji (6) The max function is an element-wise function.

4.1. Datasets

  • The Amazon5 dataset and the Amazon3 dataset contains 45, 000 training samples and 5, 000 testing samples in each class, and the samples are randomly selected from the origin data source.
  • The authors have crawled microblogs from Sina microblog website (http://weibo.com/) which has grown to be a major social media platform with hundreds of millions of users in China.
  • The authors cut off some records whose emotional tendencies are not obvious and there are 3, 000, 000 samples left.
  • The authors regard these three datasets as a benchmark to evaluate different models and explore the influence of parameters in the following experiments.

4.2. Pre-training and Word Embedding

  • There is no blank in a Chinese sentence which is different from English, so preprocessing work must be done at first to separate each sentence into several words which is called word segment and in their work the authors use an open source tool called JieBa[33] to conduct it.
  • After the word segment, the whole sentence is transformed into a sequence of Chinese words.
  • Initializing word vectors with those obtained from an unsupervised neural language model is a popular method to improve performance in the absence of a large supervised training set [34, 15, 35].
  • The authors use the publicly available word2vec tools that were trained on reviews from Amazon and SinaMicroblog for English and Chinese respectively.

4.3. Experiment Settings

  • The models are trained by min-batch back propagation with optimizer RMSprop [36] which is usually a good choice for LSTM.
  • The batch size chosen in the experiment is 128 and225 gradients are averaged over each batch.
  • Parameters of the model are randomly initialized over a uniform distribution with [-0.5, 0.5].
  • The authors set the number of kernels of convolution layers all as 200 with different window sizes and also set the number of hidden units in BLSTM as 200.
  • For regularization the authors use dropout [37] with probability 0.5 on the last Softmax layer within all models.

4.4. Results and Discussions

  • N their SICENN model, the structure of SIC is a fixed structure based on the BSLTM model.
  • The structure of SIE is more flexible.
  • Three critical factors that influence the effectiveness of SIE are explored in their following experiments.
  • The model ensemble strategy used to combine sub-extractors.

4.4.1. Size of information-extracting windows

  • In order to extract sentiment information from the sentence information matrix more240 precisely, the sizes of information-extracting windows need to be carefully chosen.
  • Resents views from amazon contains 3 categories and SinaMicroblog contain 2 categories.
  • RCNN refers to the model that Siwei proposed in [6].
  • Word embedding e(vi) is a pre-trained vector containing the semantic information of words, while sentence vectors cl(vi) and cr(vi) are the outputs of BLSTM containing the contextual information.
  • The experiments results show that265 the same window size have different performance in different datasets, which indicates the necessity to use ensemble strategy and combine the advantages of different window sizes.

4.4.2. Depth of sub-extractors

  • The depth of the sub-extractors is determined by the number of information-extracting layers, which can influence the accuracy for classification.
  • The authors have performed a series of experiments to explore how the depth of the sub-extractors influences the accuracy in the SIE.
  • Tion space than that with fewer layers, but more layers will also bring much difficulty to optimizer with backward propagation strategy.
  • The experiments results show that one layer just stands at a balance point.
  • The model ensemble strategy is essential285 for improve the performance of the information extractor and improve the accuracy for classification.

4.4.3. Model ensemble strategy

  • Model ensemble strategy can directly impact the effectiveness of the SIE and influence the results of sentiment classification.
  • Because the parameters in neural network are updated by iteration and search for the local optimal, so the initialization of these trainable parameters can influence the accuracy of sentiment classification.
  • By comparing Table 1 and Table 3, the authors can discovery that the SIE with model ensemble strategy outperforms the all the sub-extractor.
  • Besides, the SICENN model can reach a better accuracy if the authors initial the weights properly based on the results of Table 1 on different datasets.
  • The authors initial weights variables in Amazon5 as 0 ,1, 0because the extractor whose size of information extraction windows is 2 has the best performance among all the310 single window size, as is shown in Table 1.

4.5. Comparison of Methods

  • The authors compare their method with widely-used artificial neural network for sentiment anal-320 ysis including Siweis [6] model, which model has been compared with other state-of-the-art model.
  • The improvements in Amazon3 and SinaMicroblog are 0.72% and 0.73% respectively comparing their SICENN model with RCNN model.
  • The experiment results on various datasets also demonstrate their model outperforms previous state-of-the-art approaches.
  • The authors may build more sophisticated ensemble models and may involve more structures, such as attention model, to355 extract the sentiment information in the sentence more precisely.

Did you find this useful? Give us your feedback

Citations
More filters
Journal ArticleDOI
TL;DR: This paper presents a novel model for experts to carry out Group Decision Making processes using free text and alternatives pairwise comparisons and introduces two ways of applying consensus measures over the Group decision Making process.
Abstract: Social networks are the most preferred mean for the people to communicate. Therefore, it is quite usual that experts use them to carry out Group Decision Making processes. One disadvantage that recent Group Decision Making methods have is that they do not allow the experts to use free text to express themselves. On the contrary, they force them to follow a specific user–computer communication structure. This is against social network nature where experts are free to express themselves using their preferred text structure. This paper presents a novel model for experts to carry out Group Decision Making processes using free text and alternatives pairwise comparisons. The main advantage of this method is that it is designed to work using social networks. Sentiment analysis procedures are used to analyze free texts and extract the preferences that the experts provide about the alternatives. Also, our method introduces two ways of applying consensus measures over the Group Decision Making process. They can be used to determine if the experts agree among them or if there are different postures. This way, it is possible to promote the debate in those cases where consensus is low.

89 citations

Journal ArticleDOI
TL;DR: This work evaluates existing efforts proposed to do language specific sentiment analysis with a simple yet effective baseline approach and suggests that simply translating the input text in a specific language to English and then using one of the existing best methods developed for English can be better than the existing language-specific approach evaluated.

72 citations


Cites methods from "A Sentiment Information Collector-E..."

  • ...[36] uses a Bidirectional LSTM to build a Sentiment Information Collector and a Sentiment Information Extractor (SIE)....

    [...]

Journal ArticleDOI
TL;DR: In this paper, an end-to-end multi-prototype fusion embedding that fuses context-specific and task-specific information was proposed to solve the problem of polysemous-unaware word embedding.

33 citations

References
More filters
Proceedings ArticleDOI
27 Jun 2016
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

123,388 citations

Journal ArticleDOI
TL;DR: A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
Abstract: Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O. 1. Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms.

72,897 citations


"A Sentiment Information Collector-E..." refers background or methods in this paper

  • ...LSTM is developed [12] and achieved more superior performance then both tradition RNN structure and GRU [23]....

    [...]

  • ...In order to overcome the issues, LSTM was developed and achieved superior performance [12]....

    [...]

Journal ArticleDOI
01 Jan 1998
TL;DR: In this article, a graph transformer network (GTN) is proposed for handwritten character recognition, which can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters.
Abstract: Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

42,067 citations

Journal Article

28,685 citations


"A Sentiment Information Collector-E..." refers background in this paper

  • ...For RNN, gated neural networks [22] is proposed to capture the influence of the surrounding words when performing sentiment classification of entities....

    [...]

Proceedings Article
Tomas Mikolov1, Ilya Sutskever1, Kai Chen1, Greg S. Corrado1, Jeffrey Dean1 
05 Dec 2013
TL;DR: This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling.
Abstract: The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.

24,012 citations


"A Sentiment Information Collector-E..." refers background or methods in this paper

  • ...dimensionality of 300 and were trained using the continuous skip-gram architecture [11]....

    [...]

  • ...For word-embedding method, we initialize word vectors with those obtained from an unsupervised neural language model [11]....

    [...]

  • ...With the pre-trained word embeddings [9, 10, 11], neural networks demonstrate their great performance in sentiment analysis and many other NLP tasks....

    [...]

  • ...Related Work Deep learning based neural network models have achieved great success in many NLP tasks in the past few years, including learning distributed word, sentence and document representation [11], parsing [19], statistical machine translation [20], sentence classification [16, 21], etc....

    [...]