This survey paper describes a focused literature survey of machine learning (ML) and data mining (DM) methods for cyber analytics in support of intrusion detection. Short tutorial descriptions of each ML/DM method are provided. Based on the number of citations or the relevance of an emerging method, papers representing each method were identified, read, and summarized. Because data are so important in ML/DM approaches, some well-known cyber data sets used in ML/DM are described. The complexity of ML/DM algorithms is addressed, discussion of challenges for using ML/DM for cyber security is presented, and some recommendations on when to use a given method are provided.

A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection

Network Intrusion Detection Systems (NIDS) are considered as essential mechanisms to ensure reliable security. Intrusive model is used in signature-based NIDS by defining attack patterns and applying signature-matching on incoming traffic packets. Thousands of signatures and rules are created to specify different attacks and variations of a single attack. As a result, enormous data with less efficiency is produced that overwhelms the network administrator. Most of the generated alerts are false-positives; this is due to the redundancy caused by the detection techniques, and due to low-level processing capacity. Moreover, detection of novel and multi-stage attacks are not efficiently achieved by the current systems. Hence, high-level view of the attacker’s behaviour has become a stressing demand. Alerts correlation techniques have been widely used to provide intelligent and stateful detection methodologies. This is to understand attack steps and predict the expected sequence of events. However, most of the proposed systems are based on rules libraries specified by security experts, which is a cumbersome and error prone task. Other methods are based on statistical models; these are unable to identify causal relationships between the events. In this paper, we identify the limitations of the current techniques and propose a framework for alert correlation that overcomes these shortcomings. An improved “cause and effect” model will be presented cooperating with statistical model to achieve higher detection rate with minimum false positives. Knowledge-based model with vulnerability and extensional consequences parameters has been developed to provide manageable and meaningful graph. The proposed system is evaluated using DARPA 2000 and collected real life data sets. The results have shown an improvement in respect to detection rate and reduction of false positives.

MARS: Multi-stage Attack Recognition System

Intrusion detection systems (IDS) as a part of today's networks raise millions of low-level alerts every day. Consequently, it is difficult for human to analyze them. Alert correlation techniques have been developed during recent years to decrease the number of alerts and provide a high-level abstraction of them for a network administrator. In this paper, we suggest a new method for correlating alerts based on their attributes. We use time windows along with a classification method to distinguish those received alerts that are correlated with the new alert. Time windows are applied for reducing the number of comparisons and improving the accuracy of correlation. Our experiments which were done on DARPA2000 show that while the cost of comparisons dropped noticeably, the correlation method performed accurately. Our method is not limited to known attack scenarios and does not need extra domain knowledge except for training the classifier.

Alert Correlation Using Correlation Probability Estimation and Time Windows

A tremendous number of low-level alerts reported by information security systems makes it challenging for security administrators to do an effective analysis and initiate a timely response. Alert correlation techniques have been proposed to reduce the number of alerts and provide a succinct and high-level view of attacks. Most of the existing approaches rely on a priori and hard-coded domain knowledge that leads to their difficult implementation and limited capabilities of detecting new attack strategies. To address the drawbacks of these approaches, the recent trend of research in this area has gone towards extracting attack strategies through automatic analysis of intrusion alerts. In this paper, we present new algorithms to mine attack behavior patterns from a large number of intrusion alerts without specific prior knowledge about attacks. Unlike expert knowledge-based methods, our proposed scheme automatically generates correlation rules from the previously observed alerts using a Bayesian causality mechanism. The attack activity patterns learned by this way can help us to correlate alerts, reconstruct attack scenarios and predict possible forthcoming attacks in a real-time system. Our experimental results clearly show efficiency of the proposed method in learning new attack strategies.

Automatic learning of attack behavior patterns using Bayesian networks

The invention discloses a data synchronization system and a data synchronization method. A data synchronization process is finished by using a distributed peer-to-peer method, so that the data synchronization of clients does not need finishing completely in a data server and the clients can finish the data synchronization process through the coordination and management of a resource management server, which lightens the load of the data server and the response of the data server cannot be slowed down along with the increasing of the number of the clients.

Data synchronization method

Vector images are becoming more advanced and diffused on the Internet due to their unique characteristics over traditional raster images. As a W3C standard, SVG is a brand-new vector-based image format built on readable/searchable plain-text source code which inherited from XML technology, thus is suitable for Content-based image retrieval. CBIR is the application of computer vision to the image retrieval problem, which analyzes the actual contents of the images themselves rather than relying on annotations or keywords generated by human labor. Without textual descriptions or label information, successful retrieval of relevant images from large- scale image collections is a very interesting yet challenging task, so in this paper, we propose a prototype system based on SVG technology and emphasize the method of similarity measurement, also covers the implementation of the principles, which provides CBIR services.

Information Retrieval through SVG-based Vector Images Using an Original Method

The invention relates to a method for data file transmission between mobile communication equipments. The invention is to transmit data file in P2P style using mobile phone. First, user for providing uploading information divides the file into several portions or blocks, then different user downloads different block of the file, one of the user downloads a certain block of the file then uses the block for providing downloading service for other user; the method is similar to BT downloading of computer system. In the inventive method, the more people downloads, the more the servers; the more people downloads, the more people uploads; and the speed is faster.

Kaiyuan Jiang

Papers

Information Retrieval through SVG-based Vector Images Using an Original Method

Data file transmission method for mobile phone system