Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

In this paper, a hybrid method of support vector machine and genetic algorithm (GA) is proposed and its implementation in intrusion detection problem is explained. The proposed hybrid algorithm is employed in reducing the number of features from 45 to 10. The features are categorized into three priorities using GA algorithm as the highest important is the first priority and the lowest important is placed in the third priority. The feature distribution is done in a way that 4 features are placed in the first priority, 4 features in the second, and 2 features in the third priority. The results reveal that the proposed hybrid algorithm is capable of achieving a true-positive value of 0.973, while the false-positive value is 0.017.

A hybrid method consisting of GA and SVM for intrusion detection system

One of the most controversial issues in intrusion detection is automating responses to intrusions, which can provide a more efficient, quicker, and precise way to react to an attack in progress than a human. However, it comes with several disadvantages that can lead to a waste of resources, which has so far prevented wide acceptance of automated response-enabled systems. We feel that a structured approach to the problem is needed that will account for the above mentioned disadvantages. In this work, we briefly describe what has been done in the area before. Then we start addressing the problem by coupling automated response with specification-based, host-based intrusion detection. We describe the system map, and the map-based action cost model that give us the basis for deciding on response strategy. We also show the process of suspending the attack, and designing the optimal response strategy, even in the presence of uncertainty. Finally, we discuss the implementation issues, our experience with the early automated response agent prototype, the Automated Response Broker (ARB), and suggest topics for further research.

Using specification-based intrusion detection for Automated response

Rigorously characterizing the statistical properties of cyber attacks is an important problem. In this paper, we propose the first statistical framework for rigorously analyzing honeypot-captured cyber attack data. The framework is built on the novel concept of stochastic cyber attack process, a new kind of mathematical objects for describing cyber attacks. To demonstrate use of the framework, we apply it to analyze a low-interaction honeypot dataset, while noting that the framework can be equally applied to analyze high-interaction honeypot data that contains richer information about the attacks. The case study finds, for the first time, that long-range dependence (LRD) is exhibited by honeypot-captured cyber attacks. The case study confirms that by exploiting the statistical properties (LRD in this case), it is feasible to predict cyber attacks (at least in terms of attack rate) with good accuracy. This kind of prediction capability would provide sufficient early-warning time for defenders to adjust their defense configurations or resource allocations. The idea of “gray-box” (rather than “black-box”) prediction is central to the utility of the statistical framework, and represents a significant step towards ultimately understanding (the degree of) the predictability of cyber attacks.

Characterizing Honeypot-Captured Cyber Attacks: Statistical Framework and Case Study

A relatively unexplored issue in cybersecurity science and engineering is whether there exist intrinsic patterns of cyberattacks. Conventional wisdom favors absence of such patterns due to the overwhelming complexity of the modern cyberspace. Surprisingly, through a detailed analysis of an extensive data set that records the time-dependent frequencies of attacks over a relatively wide range of consecutive IP addresses, we successfully uncover intrinsic spatiotemporal patterns underlying cyberattacks, where the term “spatio” refers to the IP address space. In particular, we focus on analyzing macroscopic properties of the attack traffic flows and identify two main patterns with distinct spatiotemporal characteristics: deterministic and stochastic. Strikingly, there are very few sets of major attackers committing almost all the attacks, since their attack “fingerprints” and target selection scheme can be unequivocally identified according to the very limited number of unique spatiotemporal characteristics, each of which only exists on a consecutive IP region and differs significantly from the others. We utilize a number of quantitative measures, including the flux-fluctuation law, the Markov state transition probability matrix, and predictability measures, to characterize the attack patterns in a comprehensive manner. A general finding is that the attack patterns possess high degrees of predictability, potentially paving the way to anticipating and, consequently, mitigating or even preventing large-scale cyberattacks using macroscopic approaches.

/pdf/spatiotemporal-patterns-and-predictability-of-cyberattacks-31pji56onb.pdf

Spatiotemporal Patterns and Predictability of Cyberattacks

Monitoring Internet traffic is critical in order to acquire a good understanding of threats and in designing efficient security systems. While honeypots are flexible security tools for gathering intelligence of Internet attacks, traffic collected by honeypots is of high dimensionality that makes it difficult to characterize. In this paper, we propose the use of principal component analysis, a multivariate analysis technique, for characterizing honeypot traffic and separating latent groups of activities. In addition, we show the usefulness of principal component plots in visualizing the interrelationships between the detected groups of activities and in finding outliers. This work is demonstrated through the use of low interaction honeypot traffic data from the Leurre.com project, a world wide deployment of low interaction honeypots.

/pdf/characterization-of-attackers-activities-in-honeypot-traffic-fllko3mji2.pdf

Characterization of Attackers' Activities in Honeypot Traffic Using Principal Component Analysis

Internet Attack Knowledge Discovery Via Clusters and Cliques of Attack Traces

Honeypots are flexible security tools for gathering artefacts associated with a variety of Internet attack activities. While existing work on honeypot traffic analysis focuses mainly on identifying existing attacks, this paper describes a technique for detecting new attacks based on principal component analysis. The proposed technique requires no prior knowledge of attack types and has low computational requirements that makes it suitable for online detection systems. Our method of detecting new attacks is based on measuring changes in the residual space using square prediction error (SPE) statistics. When attack vectors are projected onto the residual space, attacks that are not presented by the main hyperspace will create new directions with high SPE values. We demonstrate the usefulness of our technique by using real traffic data from the Leurre.com project, a world-wide deployment of low-interaction honeypots, where several examples of new traffic detected by the system are illustrated.

/pdf/a-technique-for-detecting-new-attacks-in-low-interaction-rl34kz19hx.pdf

A Technique for Detecting New Attacks in Low-Interaction Honeypot Traffic

Honeypots are ﬂexible security tools for gathering artefacts associated with a variety of Internet attack activities. While existing work on honeypot trafﬁc analysis focuses mainly on identifying existing attacks, this paper describes a technique for detecting new attacks based on principal component analysis. 
The proposed technique requires no prior knowledge of attack types and has low computational requirements that makes it suitable for online detection systems. Our method of detecting new attacks is based on measuring changes in the residual space using square prediction error (SPE) statistics. When attack vectors are projected onto the residual space, attacks that are not presented by the main hyperspace will create new directions with 
high SPE values. We demonstrate the usefulness of our technique by using real trafﬁc data from the Leurre.com project, a world-wide deployment of low-interaction honeypots, where several examples of new trafﬁc detected by the system are illustrated.

A technique for detecting new attacks in low-interaction honeypot trafﬁc

Traditionally, network-based Intrusion Detection Systems (NIDS) monitor network traffic for signs of malicious activities. However, with the growing use of Virtual Private Networks (VPNs) that encrypt network traffic, the NIDS can no longer analyse the encrypted data. This essentially negates any protection offered by the NIDS. Although the encrypted traffic can be decrypted at a network gateway for analysis, this compromises on data confidentiality. In this paper, we propose a detection framework which allows a traditional NIDS to continue functioning, without compromising the confidentiality afforded by the VPN. Our approach uses Shamir's secret-sharing scheme and randomised network proxies to enable detection of malicious activities in encrypted channels. Additionally, this approach is able to detect any malicious attempts to forge network traffic with the intention of evading detection. Our experiments show that the probability of a successful evasion is low, at about 0.98\% in the worst case. We implement our approach in a prototype and present some preliminary results. Overall, the proposed approach is able to consistently detect intrusions and does not introduce any additional false positives.

Jacob Zimmermann

Papers

Characterization of Attackers' Activities in Honeypot Traffic Using Principal Component Analysis

Internet Attack Knowledge Discovery Via Clusters and Cliques of Attack Traces

A Technique for Detecting New Attacks in Low-Interaction Honeypot Traffic

A technique for detecting new attacks in low-interaction honeypot trafﬁc

Towards intrusion detection for encrypted networks