This survey paper describes a focused literature survey of machine learning (ML) and data mining (DM) methods for cyber analytics in support of intrusion detection. Short tutorial descriptions of each ML/DM method are provided. Based on the number of citations or the relevance of an emerging method, papers representing each method were identified, read, and summarized. Because data are so important in ML/DM approaches, some well-known cyber data sets used in ML/DM are described. The complexity of ML/DM algorithms is addressed, discussion of challenges for using ML/DM for cyber security is presented, and some recommendations on when to use a given method are provided.

A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection

Network Intrusion Detection Systems (NIDS) are considered as essential mechanisms to ensure reliable security. Intrusive model is used in signature-based NIDS by defining attack patterns and applying signature-matching on incoming traffic packets. Thousands of signatures and rules are created to specify different attacks and variations of a single attack. As a result, enormous data with less efficiency is produced that overwhelms the network administrator. Most of the generated alerts are false-positives; this is due to the redundancy caused by the detection techniques, and due to low-level processing capacity. Moreover, detection of novel and multi-stage attacks are not efficiently achieved by the current systems. Hence, high-level view of the attacker’s behaviour has become a stressing demand. Alerts correlation techniques have been widely used to provide intelligent and stateful detection methodologies. This is to understand attack steps and predict the expected sequence of events. However, most of the proposed systems are based on rules libraries specified by security experts, which is a cumbersome and error prone task. Other methods are based on statistical models; these are unable to identify causal relationships between the events. In this paper, we identify the limitations of the current techniques and propose a framework for alert correlation that overcomes these shortcomings. An improved “cause and effect” model will be presented cooperating with statistical model to achieve higher detection rate with minimum false positives. Knowledge-based model with vulnerability and extensional consequences parameters has been developed to provide manageable and meaningful graph. The proposed system is evaluated using DARPA 2000 and collected real life data sets. The results have shown an improvement in respect to detection rate and reduction of false positives.

MARS: Multi-stage Attack Recognition System

Intrusion detection systems (IDS) as a part of today's networks raise millions of low-level alerts every day. Consequently, it is difficult for human to analyze them. Alert correlation techniques have been developed during recent years to decrease the number of alerts and provide a high-level abstraction of them for a network administrator. In this paper, we suggest a new method for correlating alerts based on their attributes. We use time windows along with a classification method to distinguish those received alerts that are correlated with the new alert. Time windows are applied for reducing the number of comparisons and improving the accuracy of correlation. Our experiments which were done on DARPA2000 show that while the cost of comparisons dropped noticeably, the correlation method performed accurately. Our method is not limited to known attack scenarios and does not need extra domain knowledge except for training the classifier.

Alert Correlation Using Correlation Probability Estimation and Time Windows

A tremendous number of low-level alerts reported by information security systems makes it challenging for security administrators to do an effective analysis and initiate a timely response. Alert correlation techniques have been proposed to reduce the number of alerts and provide a succinct and high-level view of attacks. Most of the existing approaches rely on a priori and hard-coded domain knowledge that leads to their difficult implementation and limited capabilities of detecting new attack strategies. To address the drawbacks of these approaches, the recent trend of research in this area has gone towards extracting attack strategies through automatic analysis of intrusion alerts. In this paper, we present new algorithms to mine attack behavior patterns from a large number of intrusion alerts without specific prior knowledge about attacks. Unlike expert knowledge-based methods, our proposed scheme automatically generates correlation rules from the previously observed alerts using a Bayesian causality mechanism. The attack activity patterns learned by this way can help us to correlate alerts, reconstruct attack scenarios and predict possible forthcoming attacks in a real-time system. Our experimental results clearly show efficiency of the proposed method in learning new attack strategies.

Automatic learning of attack behavior patterns using Bayesian networks

A tremendous number of low-level alerts reported by information security systems clearly reflect the need for an advanced alert correlation system to reduce alert redundancy, correlate security alerts, detect attack strategies, and take appropriate actions against upcoming attacks. Up to now, a variety of alert correlation methods have been suggested. However, most of them rely on a priori and hard-coded domain expert knowledge that leads to their difficult implementation and limited capabilities of detecting new attack strategies. To overcome the drawbacks of these approaches, the recent trend of research in alert correlation has gone towards extracting attack strategies through automatic analysis of intrusion alerts. In line with the recent researches, in this paper, we present new algorithms to automatically mine attack behavior patterns from historical alerts as accurately and efficiently as possible. Our system is composed of two main components. The first offline component automatically generates correlation rules by analyzing the previously observed alerts using a Bayesian causality analysis mechanism. Then, in the online alert correlation component, alerts are correlated using a hierarchical scheme and based on the extracted rules. Our experimental results clearly show efficiency of the proposed method in learning new attack strategies. Copyright © 2013 John Wiley & Sons, Ltd.

A Bayesian network-based approach for learning attack strategies from intrusion alerts

Vector images are becoming more advanced and diffused on the Internet due to their unique characteristics over traditional raster images. As a W3C standard, SVG is a brand-new vector-based image format built on readable/searchable plain-text source code which inherited from XML technology, thus is suitable for Content-based image retrieval. CBIR is the application of computer vision to the image retrieval problem, which analyzes the actual contents of the images themselves rather than relying on annotations or keywords generated by human labor. Without textual descriptions or label information, successful retrieval of relevant images from large- scale image collections is a very interesting yet challenging task, so in this paper, we propose a prototype system based on SVG technology and emphasize the method of similarity measurement, also covers the implementation of the principles, which provides CBIR services.

Information Retrieval through SVG-based Vector Images Using an Original Method

In this paper, based on the Mobile E-commerce Platform, we implement the Information Retrieval System of image-based high-level semantic by using the feature extraction algorithm based on object semantic. By way of optimizing the image feature extraction algorithm, improving the structure of the traditional Search Engine and increasing the carrying capacity of mobile terminals, we will solve a core issue why Mobile E-commerce is not rife currently. It has profound effect to increase the probability of transaction, excavate potential customers and better the level of personalized service. At the same time, it satisfies the requirements of most customers.

/pdf/the-search-engine-of-image-based-high-level-semantic-2ifc4wph1q.pdf

Yu Zhou

Papers

Information Retrieval through SVG-based Vector Images Using an Original Method

The Search Engine of Image-Based High-Level Semantic Information with Application in Mobile E-Commerce