Data Mining Practical Machine Learning Tools and Techniques

Four machine learning algorithms are used for prediction in stock markets.Focus is on data pre-processing to improve the prediction accuracy.Technical indicators are discretised by exploiting the inherent opinion.Prediction accuracy of algorithms increases when discrete data is used. This paper addresses problem of predicting direction of movement of stock and stock price index for Indian stock markets. The study compares four prediction models, Artificial Neural Network (ANN), Support Vector Machine (SVM), random forest and naive-Bayes with two approaches for input to these models. The first approach for input data involves computation of ten technical parameters using stock trading data (open, high, low & close prices) while the second approach focuses on representing these technical parameters as trend deterministic data. Accuracy of each of the prediction models for each of the two input approaches is evaluated. Evaluation is carried out on 10years of historical data from 2003 to 2012 of two stocks namely Reliance Industries and Infosys Ltd. and two stock price indices CNX Nifty and S&P Bombay Stock Exchange (BSE) Sensex. The experimental results suggest that for the first approach of input data where ten technical parameters are represented as continuous values, random forest outperforms other three prediction models on overall performance. Experimental results also show that the performance of all the prediction models improve when these technical parameters are represented as trend deterministic data.

Predicting stock and stock price index movement using Trend Deterministic Data Preparation and machine learning techniques

Forecasting stock exchange rates is an important financial problem that is receiving increasing attention. During the last few years, a number of neural network models and hybrid models have been proposed for obtaining accurate prediction results, in an attempt to outperform the traditional linear and nonlinear approaches. This paper evaluates the effectiveness of neural network models which are known to be dynamic and effective in stock-market predictions. The models analysed are multi-layer perceptron (MLP), dynamic artificial neural network (DAN2) and the hybrid neural networks which use generalized autoregressive conditional heteroscedasticity (GARCH) to extract new input variables. The comparison for each model is done in two view points: Mean Square Error (MSE) and Mean Absolute Deviate (MAD) using real exchange daily rate values of NASDAQ Stock Exchange index.

Using artificial neural network models in stock market index prediction

This half-day tutorial introduces participants to data-intensive text processing with the MapReduce programming model [1], using the open-source Hadoop implementation. The focus will be on scalability and the tradeoffs associated with distributed processing of large datasets. Content will include general discussions about algorithm design, presentation of illustrative algorithms, case studies in HLT applications, as well as practical advice in writing Hadoop programs and running Hadoop clusters.

/pdf/data-intensive-text-processing-with-mapreduce-25tikv74uy.pdf

Data-Intensive Text Processing with MapReduce

Pervasive growth of Internet of Things (IoT) is visible across the globe. The 2016 Dyn cyberattack exposed the critical fault-lines among smart networks. Security of IoT has become a critical concern. The danger exposed by infested Internet-connected Things not only affects the security of IoT but also threatens the complete Internet eco-system which can possibly exploit the vulnerable Things (smart devices) deployed as botnets. Mirai malware compromised the video surveillance devices and paralyzed Internet via distributed denial of service attacks. In the recent past, security attack vectors have evolved bothways, in terms of complexity and diversity. Hence, to identify and prevent or detect novel attacks, it is important to analyze techniques in IoT context. This survey classifies the IoT security threats and challenges for IoT networks by evaluating existing defense techniques. Our main focus is on network intrusion detection systems (NIDSs); hence, this paper reviews existing NIDS implementation tools and datasets as well as free and open-source network sniffing software. Then, it surveys, analyzes, and compares state-of-the-art NIDS proposals in the IoT context in terms of architecture, detection methodologies, validation strategies, treated threats, and algorithm deployments. The review deals with both traditional and machine learning (ML) NIDS techniques and discusses future directions. In this survey, our focus is on IoT NIDS deployed via ML since learning algorithms have a good success rate in security and privacy. The survey provides a comprehensive review of NIDSs deploying different aspects of learning techniques for IoT, unlike other top surveys targeting the traditional systems. We believe that, this paper will be useful for academia and industry research, first, to identify IoT threats and challenges, second, to implement their own NIDS and finally to propose new smart techniques in IoT context considering IoT limitations. Moreover, the survey will enable security individuals differentiate IoT NIDS from traditional ones.

Network Intrusion Detection for IoT Security Based on Learning Techniques

In this paper we propose and implement a fusion model by combining the Hidden Markov Model (HMM), Artificial Neural Networks (ANN) and Genetic Algorithms (GA) to forecast financial market behaviour. The developed tool can be used for in depth analysis of the stock market. Using ANN, the daily stock prices are transformed to independent sets of values that become input to HMM. We draw on GA to optimize the initial parameters of HMM. The trained HMM is used to identify and locate similar patterns in the historical data. The price differences between the matched days and the respective next day are calculated. Finally, a weighted average of the price differences of similar patterns is obtained to prepare a forecast for the required next day. Forecasts are obtained for a number of securities in the IT sector and are compared with a conventional forecast method.

A fusion model of HMM, ANN and GA for stock market forecasting

This paper presents hidden Markov models (HMM) approach for forecasting stock price for interrelated markets. We apply HMM to forecast some of the airlines stock. HMMs have been extensively used for pattern recognition and classification problems because of its proven suitability for modelling dynamic systems. However, using HMM for predicting future events is not straightforward. Here we use only one HMM that is trained on the past dataset of the chosen airlines. The trained HMM is used to search for the variable of interest behavioural data pattern from the past dataset. By interpolating the neighbouring values of these datasets forecasts are prepared. The results obtained using HMM are encouraging and HMM offers a new paradigm for stock market forecasting, an area that has been of much research interest lately.

http://mleg.cse.sc.edu/edu/csce768/uploads/Main.ReadingList/HMM-stock.pdf

Stock market forecasting using hidden Markov model: a new approach

Intrusion detection faces a number of challenges; an intrusion detection system must reliably detect malicious activities in a network and must perform efficiently to cope with the large amount of network traffic. In this paper, we address these two issues of Accuracy and Efficiency using Conditional Random Fields and Layered Approach. We demonstrate that high attack detection accuracy can be achieved by using Conditional Random Fields and high efficiency by implementing the Layered Approach. Experimental results on the benchmark KDD '99 intrusion data set show that our proposed system based on Layered Conditional Random Fields outperforms other well-known methods such as the decision trees and the naive Bayes. The improvement in attack detection accuracy is very high, particularly, for the U2R attacks (34.8 percent improvement) and the R2L attacks (34.5 percent improvement). Statistical Tests also demonstrate higher confidence in detection accuracy for our method. Finally, we show that our system is robust and is able to handle noisy data without compromising performance.

Layered Approach Using Conditional Random Fields for Intrusion Detection

Identifying the vascular bifurcations and crossovers in the retinal image is helpful for predicting many cardiovascular diseases and can be used as biometric features and for image registration. In this paper, we propose an efficient method to detect vascular bifurcations and crossovers based on the vessel geometrical features. We segment the blood vessels from the color retinal RGB image, and apply the morphological thinning operation to find the vessel centerline. Applying a filter on this centreline image we detect the potential bifurcation and crossover points. The geometrical and topological properties of the blood vessels passing through these points are utilized to identify these points as the vessel bifurcations and crossovers. We evaluate our method against manually measured bifurcation and crossover points by an expert, and achieved the detection accuracy of 95.82%.

Automatic Detection of Vascular Bifurcations and Crossovers from Color Retinal Fundus Images

Automated blood vessel segmentation is an important issue for assessing retinal abnormalities and diagnoses of many diseases. The segmentation of vessels is complicated by huge variations in local contrast, particularly in case of the minor vessels. In this paper, we propose a new method of texture based vessel segmentation to overcome this problem. We use Gaussian and L*a*b* perceptually uniform color spaces with original RGB for texture feature extraction on retinal images. A bank of Gabor energy filters are used to analyze the texture features from which a feature vector is constructed for each pixel. The fuzzy C-means (FCM) clustering algorithm is used to classify the feature vectors into vessel or non-vessel based on the texture properties. From the FCM clustering output we attain the final output segmented image after a post processing step. We compare our method with hand-labeled ground truth segmentation of five images and achieve 84.37% sensitivity and 99.61% specificity.

Baikunth Nath

Papers

A fusion model of HMM, ANN and GA for stock market forecasting

Stock market forecasting using hidden Markov model: a new approach

Layered Approach Using Conditional Random Fields for Intrusion Detection

Automatic Detection of Vascular Bifurcations and Crossovers from Color Retinal Fundus Images

Blood Vessel Segmentation from Color Retinal Images using Unsupervised Texture Classification