This survey paper describes a focused literature survey of machine learning (ML) and data mining (DM) methods for cyber analytics in support of intrusion detection. Short tutorial descriptions of each ML/DM method are provided. Based on the number of citations or the relevance of an emerging method, papers representing each method were identified, read, and summarized. Because data are so important in ML/DM approaches, some well-known cyber data sets used in ML/DM are described. The complexity of ML/DM algorithms is addressed, discussion of challenges for using ML/DM for cyber security is presented, and some recommendations on when to use a given method are provided.

A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection

Network anomaly detection is an important and dynamic research area. Many network intrusion detection methods and systems (NIDS) have been proposed in the literature. In this paper, we provide a structured and comprehensive overview of various facets of network anomaly detection so that a researcher can become quickly familiar with every aspect of network anomaly detection. We present attacks normally encountered by network intrusion detection systems. We categorize existing network anomaly detection methods and systems based on the underlying computational techniques used. Within this framework, we briefly describe and compare a large number of network anomaly detection methods and systems. In addition, we also discuss tools that can be used by network defenders and datasets that researchers in network anomaly detection can use. We also highlight research directions in network anomaly detection.

Network Anomaly Detection: Methods, Systems and Tools

Machine Learning (ML) has been enjoying an unprecedented surge in applications that solve problems and enable automation in diverse domains. Primarily, this is due to the explosion in the availability of data, significant improvements in ML techniques, and advancement in computing capabilities. Undoubtedly, ML has been applied to various mundane and complex problems arising in network operation and management. There are various surveys on ML for specific areas in networking or for specific network technologies. This survey is original, since it jointly presents the application of diverse ML techniques in various key areas of networking across different network technologies. In this way, readers will benefit from a comprehensive discussion on the different learning paradigms and ML techniques applied to fundamental problems in networking, including traffic prediction, routing and classification, congestion control, resource and fault management, QoS and QoE management, and network security. Furthermore, this survey delineates the limitations, give insights, research challenges and future opportunities to advance ML in networking. Therefore, this is a timely contribution of the implications of ML for networking, that is pushing the barriers of autonomic network operation and management.

/pdf/a-comprehensive-survey-on-machine-learning-for-networking-3t5h3kbpe8.pdf

A comprehensive survey on machine learning for networking: evolution, applications and research opportunities

The Internet of Things (IoT) integrates billions of smart devices that can communicate with one another with minimal human intervention. IoT is one of the fastest developing fields in the history of computing, with an estimated 50 billion devices by the end of 2020. However, the crosscutting nature of IoT systems and the multidisciplinary components involved in the deployment of such systems have introduced new security challenges. Implementing security measures, such as encryption, authentication, access control, network and application security for IoT devices and their inherent vulnerabilities is ineffective. Therefore, existing security methods should be enhanced to effectively secure the IoT ecosystem. Machine learning and deep learning (ML/DL) have advanced considerably over the last few years, and machine intelligence has transitioned from laboratory novelty to practical machinery in several important applications. Consequently, ML/DL methods are important in transforming the security of IoT systems from merely facilitating secure communication between devices to security-based intelligence systems. The goal of this work is to provide a comprehensive survey of ML methods and recent advances in DL methods that can be used to develop enhanced security methods for IoT systems. IoT security threats that are related to inherent or newly introduced threats are presented, and various potential IoT system attack surfaces and the possible threats related to each surface are discussed. We then thoroughly review ML/DL methods for IoT security and present the opportunities, advantages and shortcomings of each method. We discuss the opportunities and challenges involved in applying ML/DL to IoT security. These opportunities and challenges can serve as potential future research directions.

A Survey of Machine and Deep Learning Methods for Internet of Things (IoT) Security

A blockchain future for internet of things security: a position paper

The IT community is confronted with incidents of all kinds and nature, new threats appear on a daily basis. Fighting these security incidents individually is almost impossible. Sharing information about threats among the community has become a key element in incident response to stay on top of the attackers. Reliable information resources, providing credible information, are therefore essential to the IT community, or even at broader scale, to intelligence communities or fraud detection groups. This paper presents the Malware Information Sharing Platform (MISP) and threat sharing project, a trusted platform, that allows the collection and sharing of important indicators of compromise (IoC) of targeted attacks, but also threat information like vulnerabilities or financial indicators used in fraud cases. The aim of MISP is to help in setting up preventive actions and counter-measures used against targeted attacks. Enable detection via collaborative-knowledge-sharing about existing malware and other threats.

MISP: The Design and Implementation of a Collaborative Threat Intelligence Sharing Platform

Faced to continuous arising new threats, the detection of anomalies in current operational networks has become essential. Network operators have to deal with huge data volumes for analysis purpose. To counter this main issue, dealing with IP flow (also known as Netflow) records is common in network management. However, still in modern networks, Netflow records represent high volume of data. In this paper, we present an approach for evaluating Netflow records by referring to a method of temporal aggregation applied to Machine Learning techniques. We present an approach that leverages support vector machines in order to analyze large volumes of Netflow records. Our approach is using a special kernel function, that takes into account both the contextual and the quantitative information of Netflow records. We assess the viability of our method by practical experimentation on data volumes provided by a major internet service provider in Luxembourg.

/pdf/machine-learning-approach-for-ip-flow-record-anomaly-1x7mc03syl.pdf

Machine learning approach for IP-flow record anomaly detection

This paper addresses a fundamentally new method for analyzing the behavior of executed applications and sessions. We describe a modeling framework capable of representing relationships among processes belonging to the same session in an integrated way, as well as the information related to the underlying system calls executed. We leverage for this purpose graph-based kernels and Support Vector Machines (SVM) in order to classify either individually monitored applications or more comprehensive user sessions. Our approach can serve both as a host-level intrusion detection and application level monitoring and as an adaptive jail framework.

Malware analysis with graph kernels and support vector machines

We present a monitoring approach and the supporting software architecture for passive DNS traffic. Monitoring DNS traffic can reveal essential network and system level activity profiles. Worm infected and botnet participating hosts can be identified and malicious backdoor communications can be detected. Any passive DNS monitoring solution needs to address several challenges that range from architectural approaches for dealing with large volumes of data up to specific Data Mining approaches for this purpose. We describe a framework that leverages state of the art distributed processing facilities with clustering techniques in order to detect anomalies in both online and offline DNS traffic. This framework entitled DNSSM is implemented and operational on several networks. We validate the framework against two large trace sets1.

/pdf/dnssm-a-large-scale-passive-dns-security-monitoring-1fmgjs5vyp.pdf

DNSSM: A large scale passive DNS security monitoring framework

The Internet has grown into an enormous network offering a variety of services, which are spread over a multitude of domains. BGP-routing and Autonomous Systems (AS) are the key components for maintaining high connectivity in the Internet. Unfortunately, Internet Service Providers (ISPs) operating ASs do not only host normal users and content, but also malicious content used by attackers for spreading malware, hosting phishing websites or performing any kind of fraudulent activity. Practical analysis shows that such malware-providing ASs prevent themselves from being de-peered by hiding behind other ASs, which do not host the malware themselves but simply provide transit service for malware. This paper presents a new method for detecting ASs that provide transit service for malware hosters, without being malicious themselves. A formal definition of the problem and the metrics are determined by using the AS graph. The PageRank algorithm is applied to improve the scalability and the completeness of the approach. The method is assessed on real and publicly available datasets, showing promising results.

https://hal.inria.fr/hal-00846082/document

Cynthia Wagner

Papers

MISP: The Design and Implementation of a Collaborative Threat Intelligence Sharing Platform

Machine learning approach for IP-flow record anomaly detection

Malware analysis with graph kernels and support vector machines

DNSSM: A large scale passive DNS security monitoring framework

ASMATRA: Ranking ASs providing transit service to malware hosters