Online information has become important data source to analyze the public opinion and behavior, which is significant for social management and business decision. Web crawler systems target at automatically download and parse web pages to extract expected online information. However, as the rapid increasing of web pages and the heterogeneous page structures, the performance and the rules of parsing have become two serious challenges to web crawler systems. In this paper, we propose a distributed and generic web crawler system (DGWC), in which spiders are scheduled to parallel access and parse web pages to improve performance, utilized a shared and memory based database. Furthermore, we package the spider program and the dependencies in a container called Docker to make the system easily horizontal scaling. Last but not the least, a statistics-based approach is proposed to extract the main text using supervised-learning classifier instead of parsing the page structures. Experimental results on real-world data validate the efficiency and effectiveness of DGWC.

DGWC: Distributed and generic web crawler for online information extraction

Online Public Opinion Systems (OPOS) target at collecting, analyzing, summarizing and monitoring massive public opinions on the Internet in real time. Meanwhile, OPOS often have the ability to identify the key or sudden events, and thus notify related people immediately for rapid responses to these events. As part of this endeavor, this paper introduces the architecture and techniques of an OPOS that has been used by several large enterprises. This self-designed OPOS generally contains data layer, computation layer and application layer from bottom to up. We first introduce the basic function and key techniques of each layer, and then present several typical yet important algorithms on the computation layer. Experimental results on real-world data validate the effectiveness of algorithms fixed in our system. Last but not the least, a system demonstration in a ship-building company is provided to justify the value of our OPOS for real enterprises.

Online Public Opinion System: Design and Applications

Unconventional emergency is a typical complex system which has characteristics of diversity,randomness,suddeness and disorder.According to the complexity of emergency decision-making of unconventional emergency,a model of evaluation and decision-making for complexity measurement based on information entropy was proposed.It can effectively choose the best decision in the decision-making and evaluation system.Experiments show that the method of complexity measurement can help select the better solution.

Application of Complexity Measurement in the Unconventional for Decision-making and Evaluation

This paper investigates the emergency group decision‐making problem based on interval‐valued Pythagorean fuzzy language sets (IVPFLSs). The frequent occurrence of emergency events can bring huge economic damage to human beings. To reduce the loss, it is very important to make reasonable emergency decisions effectively and timely. In the emergency decision making (EDM), these problems are few studied, such as high dimension problem, data nonlinearity, and correlation. For EDM problems, the advantage of IVPFLSs is that it can reasonably express the evaluation information given by decision makers (DMs) through both qualitative and quantitative aspects. However, if the dimension and nonlinear relationship of the decision data keep growing, and the traditional decision‐making methods will fail. The distance measure between decision data is necessary to calculate in the process of dimensionality reduction, and the current research does not propose the definition of IVPFLSs distance measure. On the basis of this, this paper first uses the attributes and DMs as variables to define the standard Euclidean distance measure between IVPFLSs. For nonlinear features, we construct the interval‐valued Pythagorean fuzzy language kernel principal component analysis (IVPFL‐KPCA) model to reduce the dimensionality. What is more, we also obtain the reasonable weight vectors of the attribute and DMs from cumulative contribution rate. For low‐dimensional decision data, according to the technique for order performance by similarity to ideal solution method, the best emergency plan is selected for the information variables after dimensionality reduction. In sum, the IVPFL‐KPCA model not only avoids multicollinearity and nonlinear separability between decision data, but also obtains reasonable weights. It further improves the efficiency of decision‐making operations and reduces the difficulty of the algorithm. Finally, an example of earthquake emergency plan is shown to demonstrate the feasibility and practicability of the proposed method. Besides, we also compare it with the existing methods, which proves the effectiveness of the method.

Interval‐valued Pythagorean fuzzy linguistic KPCA model based on TOPSIS and its application for emergency group decision making

With the rapid development of mobile Internet, social network has become an important platform for users to obtain information, express opinions and exchange views. Agriculture concerns people's livelihood, and the network conveys public opinion. It is urgent to improve the monitoring ability of agricultural network public opinion by correctly guiding the guidance of agricultural public opinion and preventing the spread of negative public opinion. At the same time, it puts forward higher requirements for network public opinion monitoring. Using the massive data processing performance of text mining to solve the problems of low storage space expansion and computational efficiency of agricultural-related network public opinion monitoring system, it provides theoretical support and decision-making basis for workers and decision-makers in government, enterprise and other fields. Around the design and implementation of the network public opinion supervision and management system, it has realized the functions of public opinion collection, public opinion analysis and tracking, emotional orientation judgment and public opinion warning. The sentiment orientation analysis model is constructed to judge the emotional type and statistical sentiment frequency of the online public opinion, and to mine and visually analyze the user emotion in the event. The system is operating normally and can be effectively monitored to meet design goals.

Early Warning Mechanism of Agricultural Network Public Opinion Based on Text Mining

In order to facilitate government making timely decision for multiple internet public opinion emergencies with interval fuzzy indicators, we first determine the rational weight of each emergency decision index based on fuzzy information entropy of interval index. Then we aggregate all the interval index values to obtain the comprehensive alarm interval of each emergency. Finally, we can get the priority of emergency operation of all the internet public opinion emergencies according to the possible degree between their alarm intervals.

Uncertain Internet Public Opinion Emergency Decision-Making Method under Interval-Valued Fuzzy Environment

In recent years, online social networks (OSNs) have become great places for people to communicate with each other and share knowledge. However, OSNs have also become the main grounds for exploiting the vulnerabilities of people and launching a variety of fraud. Most of hackers implement fraud based on the target users’ personality traits. It is difficult for users to identify such fraud in OSNs. In order to alert users to the hackers’ fraud strategies, and improve users’ ability to identify fraud, the research on personality trait identification is important. In this paper a new method is presented for identifying user's personality trait based on the personality trait dictionary and hidden semi-Markov models, from the perspective of the behavior process of user's posting/forwarding information in OSNs. The proposed method includes a training phase and an identification phase. In the identification phase, the average log likelihood of every observation sequence is calculated. An experiment based on real datasets of Weibo is conducted to evaluate this method. The experiment results validate the effectiveness of this method.

Personality Trait Identification Based on Hidden semi-Markov Model in Online Social Networks

EDRN-based propagation model for popular microblog information detection

Detecting fake news by RNN-based gatekeeping behavior model on social networks

Nowadays, a large number of people frequently use social networks. Social networks have become important platforms for people to publish and obtain information. However, social networks have also become the main venue for hackers to initiate online fraud. Phishing is a common way used by hackers to launch online fraud on social networks. This paper proposes a method for detecting phishing short URL based on the link jumping. The method uses a hierarchical hidden Markov model with two-layer structure to describe the link jumping process after user clicking on short URL, so as to identify phishing short URL on social networks. The proposed method includes a training phase and an identification phase. In the identification phase, the average log-likelihood probability of the observation sequence is calculated. An experiment based on real datasets of Weibo is conducted to evaluate this method. The experiment results validate the effectiveness of this method.

Bai Lin Xie

Papers

Uncertain Internet Public Opinion Emergency Decision-Making Method under Interval-Valued Fuzzy Environment

Personality Trait Identification Based on Hidden semi-Markov Model in Online Social Networks

EDRN-based propagation model for popular microblog information detection

Detecting fake news by RNN-based gatekeeping behavior model on social networks

Phishing short URL detection based on link jumping on social networks