Predicting the direction of stock market prices using tree-based classifiers

The stock market is a key pivot in every growing and thriving economy, and every investment in the market is aimed at maximising profit and minimising associated risk. As a result, numerous studies have been conducted on the stock-market prediction using technical or fundamental analysis through various soft-computing techniques and algorithms. This study attempted to undertake a systematic and critical review of about one hundred and twenty-two (122) pertinent research works reported in academic journals over 11 years (2007–2018) in the area of stock market prediction using machine learning. The various techniques identified from these reports were clustered into three categories, namely technical, fundamental, and combined analyses. The grouping was done based on the following criteria: the nature of a dataset and the number of data sources used, the data timeframe, the machine learning algorithms used, machine learning task, used accuracy and error metrics and software packages used for modelling. The results revealed that 66% of documents reviewed were based on technical analysis; whiles 23% and 11% were based on fundamental analysis and combined analyses, respectively. Concerning the number of data source, 89.34% of documents reviewed, used single sources; whiles 8.2% and 2.46% used two and three sources respectively. Support vector machine and artificial neural network were found to be the most used machine learning algorithms for stock market prediction.

A systematic review of fundamental and technical analysis of stock market predictions

The new transformed read-write Web has resulted in a rapid growth of user generated content on the Web resulting into a huge volume of unstructured data. A substantial part of this data is unstructured text such as reviews and blogs. Opinion mining and sentiment analysis (OMSA) as a research discipline has emerged during last 15 years and provides a methodology to computationally process the unstructured data mainly to extract opinions and identify their sentiments. The relatively new but fast growing research discipline has changed a lot during these years. This paper presents a scientometric analysis of research work done on OMSA during 2000–2016. For the scientometric mapping, research publications indexed in Web of Science (WoS) database are used as input data. The publication data is analyzed computationally to identify year-wise publication pattern, rate of growth of publications, types of authorship of papers on OMSA, collaboration patterns in publications on OMSA, most productive countries, institutions, journals and authors, citation patterns and an year-wise citation reference network, and theme density plots and keyword bursts in OMSA publications during the period. A somewhat detailed manual analysis of the data is also performed to identify popular approaches (machine learning and lexicon-based) used in these publications, levels (document, sentence or aspect-level) of sentiment analysis work done and major application areas of OMSA. The paper presents a detailed analytical mapping of OMSA research work and charts the progress of discipline on various useful parameters.

Analytical mapping of opinion mining and sentiment analysis research during 2000–2015

The purpose of this paper is to develop a theoretical model to explain the impact of big data and predictive analytics (BDPA) on sustainable business development goal of the organization.,The authors have developed the theoretical model using resource-based view logic and contingency theory. The model was further tested using partial least squares-structural equation modeling (PLS-SEM) following Peng and Lai (2012) arguments. The authors gathered 205 responses using survey-based instrument for PLS-SEM.,The statistical results suggest that out of four research hypotheses, the authors found support for three hypotheses (H1-H3) and the authors did not find support for H4. Although the authors did not find support for H4 (moderating role of supply base complexity (SBC)), however, in future the relationship between BDPA, SBC and sustainable supply chain performance measures remain interesting research questions for further studies.,This study makes some original contribution to the operations and supply chain management literature. The authors provide theory-driven and empirically proven results which extend previous studies which have focused on single performance measures (i.e. economic or environmental). Hence, by studying the impact of BDPA on three performance measures the authors have attempted to answer some of the unresolved questions. The authors also offer numerous guidance to the practitioners and policy makers, based on empirical results.

/pdf/impact-of-big-data-predictive-analytics-capability-on-supply-9szann1ng3.pdf

Impact of big data & predictive analytics capability on supply chain sustainability

The development of IoT technologies and the massive admiration and acceptance of social media tools and applications, new doors of opportunity have been opened for using data analytics in gaining meaningful insights from unstructured information. The application of opinion mining and sentiment analysis (OMSA) in the era of big data have been used a useful way in categorizing the opinion into different sentiment and in general evaluating the mood of the public. Moreover, different techniques of OMSA have been developed over the years in different data sets and applied to various experimental settings. In this regard, this paper presents a comprehensive systematic literature review, aims to discuss both technical aspect of OMSA (techniques and types) and non-technical aspect in the form of application areas are discussed. Furthermore, this paper also highlighted both technical aspects of OMSA in the form of challenges in the development of its technique and non-technical challenges mainly based on its application. These challenges are presented as a future direction for research.

Sentiment Analysis of Big Data: Methods, Applications, and Open Challenges

This is a known fact that news and stock prices are closely related and news usually has a great influence on stock market investment. There have been many researches aimed at identifying that relationship or predicting stock market movements using news analysis. Recently, massive news tests, called unstructured big-data, have been used to predict stock price. In this paper, we introduce a method of mining text opinions to analyze Korean language news in order to predict rises and falls on the KOSPI (Korea Composite Stock Price Index). Our method consists of carrying out the NLP (Natural Language Processing) of news, describing its features, categorizing and extracting the sentiments and opinions expressed by the writers. The method then identifies the correlation between news and stock market fluctuations. In our experiment, we show that our method can be used to understand unstructured big-data, and we also reveal that news’ sentiment can be used in predicting stock price fluctuations, whether up or down. The algorithm extracted experiments can be used to make predictions about stock market movements .

http://home.ijasca.com/data/documents/Paper-ID-424-IJASCA_Formated.pdf

Text Opinion Mining to Analyze News for Stock Market Prediction

Purpose – The purpose of this paper is to mine competitive intelligence in social media to find the market insight by comparing consumer opinions and sales performance of a business and one of its competitors by analyzing the public social media data. Design/methodology/approach – An exploratory test using a multiple case study approach was used to compare two competing smartphone manufacturers. Opinion mining and sentiment analysis are conducted first, followed by further validation of results using statistical analysis. A total of 229,948 tweets mentioning the iPhone6 or the GalaxyS5 have been collected for four months following the release of the iPhone6; these have been analyzed using natural language processing, lexicon-based sentiment analysis, and purchase intention classification. Findings – The analysis showed that social media data contain competitive intelligence. The volume of tweets revealed a significant gap between the market leader and one follower; the purchase intention data also reflect...

Competitive intelligence in social media Twitter: iPhone 6 vs. Galaxy S5

Recently, the amount of unstructured data being generated through a variety of social media has been increasing rapidly, resulting in the increasing need to collect, store, search for, analyze, and visualize this data. This kind of data cannot be handled appropriately by using the traditional methodologies usually used for analyzing structured data because of its vast volume and unstructured nature. In this situation, many attempts are being made to analyze unstructured data such as text files and log files through various commercial or noncommercial analytical tools. Among the various contemporary issues dealt with in the literature of unstructured text data analysis, the concepts and techniques of opinion mining have been attracting much attention from pioneer researchers and business practitioners. Opinion mining or sentiment analysis refers to a series of processes that analyze participants` opinions, sentiments, evaluations, attitudes, and emotions about selected products, services, organizations, social issues, and so on. In other words, many attempts based on various opinion mining techniques are being made to resolve complicated issues that could not have otherwise been solved by existing traditional approaches. One of the most representative attempts using the opinion mining technique may be the recent research that proposed an intelligent model for predicting the direction of the stock index. This model works mainly on the basis of opinions extracted from an overwhelming number of economic news repots. News content published on various media is obviously a traditional example of unstructured text data. Every day, a large volume of new content is created, digitalized, and subsequently distributed to us via online or offline channels. Many studies have revealed that we make better decisions on political, economic, and social issues by analyzing news and other related information. In this sense, we expect to predict the fluctuation of stock markets partly by analyzing the relationship between economic news reports and the pattern of stock prices. So far, in the literature on opinion mining, most studies including ours have utilized a sentiment dictionary to elicit sentiment polarity or sentiment value from a large number of documents. A sentiment dictionary consists of pairs of selected words and their sentiment values. Sentiment classifiers refer to the dictionary to formulate the sentiment polarity of words, sentences in a document, and the whole document. However, most traditional approaches have common limitations in that they do not consider the flexibility of sentiment polarity, that is, the sentiment polarity or sentiment value of a word is fixed and cannot be changed in a traditional sentiment dictionary. In the real world, however, the sentiment polarity of a word can vary depending on the time, situation, and purpose of the analysis. It can also be contradictory in nature. The flexibility of sentiment polarity motivated us to conduct this study. In this paper, we have stated that sentiment polarity should be assigned, not merely on the basis of the inherent meaning of a word but on the basis of its ad hoc meaning within a particular context. To implement our idea, we presented an intelligent investment decision-support model based on opinion mining that performs the scrapping and parsing of massive volumes of economic news on the web, tags sentiment words, classifies sentiment polarity of the news, and finally predicts the direction of the next day`s stock index. In addition, we applied a domain-specific sentiment dictionary instead of a general purpose one to classify each piece of news as either positive or negative. For the purpose of performance evaluation, we performed intensive experiments and investigated the prediction accuracy of our model. For the experiments to predict the direction of the stock index, we gathered and analyzed 1,072 articles about stock markets published by "M" and "E" media between July 2011 and September 2011.

/pdf/predicting-the-direction-of-the-stock-index-by-using-a-49qqbau6q6.pdf

Predicting the Direction of the Stock Index by Using a Domain-Specific Sentiment Dictionary

Social media have emerged as new communication channels between consumers and companies that generate a large volume of unstructured text data. This social media content, which contains consumers’ opinions and interests, is recognized as valuable material from which businesses can mine useful information; consequently, many researchers have reported on opinion-mining frameworks, methods, techniques, and tools for business intelligence over various industries. These studies sometimes focused on how to use opinion mining in business fields or emphasized methods of analyzing content to achieve results that are more accurate. They also considered how to visualize the results to ensure easier understanding. However, we found that such approaches are often technically complex and insufficiently user-friendly to help with business decisions and planning. Therefore, in this study we attempt to formulate a more comprehensive and practical methodology to conduct social media opinion mining and apply our methodology to a case study of the oldest instant noodle product in Korea. We also present graphical tools and visualized outputs that include volume and sentiment graphs, time-series graphs, a topic word cloud, a heat map, and a valence tree map with a classification. Our resources are from public-domain social media content such as blogs, forum messages, and news articles that we analyze with natural language processing, statistics, and graphics packages in the freeware R project environment. We believe our methodology and visualization outputs can provide a practical and reliable guide for immediate use, not just in the food industry but other industries as well.

Opinion-Mining Methodology for Social Media Analytics

People easily believe that news and stock index are closely related. They think that securing news before anyone else can help them forecast the stock prices and enjoy great profit, or perhaps capture the investment opportunity. However, it is no easy feat to determine to what extent the two are related, come up with the investment decision based on news, or find out such investment information is valid. If the significance of news and its impact on the stock market are analyzed, it will be possible to extract the information that can assist the investment decisions. The reality however is that the world is inundated with a massive wave of news in real time. And news is not patterned text. This study suggests the stock-index invest model based on "News Big Data" opinion mining that systematically collects, categorizes and analyzes the news and creates investment information. To verify the validity of the model, the relationship between the result of news opinion mining and stock-index was empirically analyzed by using statistics. Steps in the mining that converts news into information for investment decision making, are as follows. First, it is indexing information of news after getting a supply of news from news provider that collects news on real-time basis. Not only contents of news but also various information such as media, time, and news type and so on are collected and classified, and then are reworked as variable from which investment decision making can be inferred. Next step is to derive word that can judge polarity by separating text of news contents into morpheme, and to tag positive/negative polarity of each word by comparing this with sentimental dictionary. Third, positive/negative polarity of news is judged by using indexed classification information and scoring rule, and then final investment decision making information is derived according to daily scoring criteria. For this study, KOSPI index and its fluctuation range has been collected for 63 days that stock market was open during 3 months from July 2011 to September in Korea Exchange, and news data was collected by parsing 766 articles of economic news media M company on web page among article carried on stock information>news>main news of portal site Naver.com. In change of the price index of stocks during 3 months, it rose on 33 days and fell on 30 days, and news contents included 197 news articles before opening of stock market, 385 news articles during the session, 184 news articles after closing of market. Results of mining of collected news contents and of comparison with stock price showed that positive/negative opinion of news contents had significant relation with stock price, and change of the price index of stocks could be better explained in case of applying news opinion by deriving in positive/negative ratio instead of judging between simplified positive and negative opinion. And in order to check whether news had an effect on fluctuation of stock price, or at least went ahead of fluctuation of stock price, in the results that change of stock price was compared only with news happening before opening of stock market, it was verified to be statistically significant as well. In addition, because news contained various type and information such as social, economic, and overseas news, and corporate earnings, the present condition of type of industry, market outlook, the present condition of market and so on, it was expected that influence on stock market or significance of the relation would be different according to the type of news, and therefore each type of news was compared with fluctuation of stock price, and the results showed that market condition, outlook, and overseas news was the most useful to explain fluctuation of news. On the contrary, news about individual company was not statistically significant, but opinion mining value showed tendency opposite to stock price, and the reason can be thought to be the appearance of promotional and planned news for preventing stock price from falling. Finally, multiple regression analysis and logistic regression analysis was carried out in order to derive function of investment decision making on the basis of relation between positive/negative opinion of news and stock price, and the results showed that regression equation using variable of market conditions, outlook, and overseas news before opening of stock market was statistically significant, and classification accuracy of logistic regression accuracy results was shown to be 70.0% in rise of stock price, 78.8% in fall of stock price, and 74.6% on average. This study first analyzed relation between news and stock price through analyzing and quantifying sensitivity of atypical news contents by using opinion mining among big data analysis techniques, and furthermore, proposed and verified smart investment decision making model that could systematically carry out opinion mining and derive and support investment information. This shows that news can be used as variable to predict the price index of stocks for investment, and it is expected the model can be used as real investment support system if it is implemented as system and verified in the future.

https://www.koreascience.or.kr:443/article/JAKO201223052004289.pdf

Yoosin Kim

Papers

Text Opinion Mining to Analyze News for Stock Market Prediction

Competitive intelligence in social media Twitter: iPhone 6 vs. Galaxy S5

Predicting the Direction of the Stock Index by Using a Domain-Specific Sentiment Dictionary

Opinion-Mining Methodology for Social Media Analytics

Stock-Index Invest Model Using News Big Data Opinion Mining