Network Traffic Monitoring and Analysis (NTMA) represents a key component for network management, especially to guarantee the correct operation of large-scale networks such as the Internet. As the complexity of Internet services and the volume of traffic continue to increase, it becomes difficult to design scalable NTMA applications. Applications such as traffic classification and policing require real-time and scalable approaches. Anomaly detection and security mechanisms require to quickly identify and react to unpredictable events while processing millions of heterogeneous events. At last, the system has to collect, store, and process massive sets of historical data for post-mortem analysis. Those are precisely the challenges faced by general big data approaches: Volume, Velocity, Variety, and Veracity. This survey brings together NTMA and big data. We catalog previous work on NTMA that adopt big data approaches to understand to what extent the potential of big data is being explored in NTMA. This survey mainly focuses on approaches and technologies to manage the big NTMA data, additionally briefly discussing big data analytics (e.g., machine learning) for the sake of NTMA. Finally, we provide guidelines for future work, discussing lessons learned, and research directions.

A Survey on Big Data for Network Traffic Monitoring and Analysis

Detecting and monitoring of abnormal movement behaviors in patients with Parkinson’s Disease (PD) and individuals with Autism Spectrum Disorders (ASD) are beneficial for adjusting care and medical treatment in order to improve the patient’s quality of life. Supervised methods commonly used in the literature need annotation of data, which is a time-consuming and costly process. In this paper, we propose deep normative modeling as a probabilistic novelty detection method, in which we model the distribution of normal human movements recorded by wearable sensors and try to detect abnormal movements in patients with PD and ASD in a novelty detection framework. In the proposed deep normative model, a movement disorder behavior is treated as an extreme of the normal range or, equivalently, as a deviation from the normal movements. Our experiments on three benchmark datasets indicate the effectiveness of the proposed method, which outperforms one-class SVM and the reconstruction-based novelty detection approaches. Our contribution opens the door toward modeling normal human movements during daily activities using wearable sensors and eventually real-time abnormal movement detection in neuro-developmental and neuro-degenerative disorders.

Novelty Detection using Deep Normative Modeling for IMU-Based Abnormal Movement Monitoring in Parkinson's Disease and Autism Spectrum Disorders.

Cyber situational awareness has been proven to be of value in forming a comprehensive understanding of threats and vulnerabilities within organisations, as the degree of exposure is governed by the prevailing levels of cyber-hygiene and established processes. A more accurate assessment of the security provision informs on the most vulnerable environments that necessitate more diligent management. The rapid proliferation in the automation of cyber-attacks is reducing the gap between information and operational technologies and the need to review the current levels of robustness against new sophisticated cyber-attacks, trends, technologies and mitigation countermeasures has become pressing. A deeper characterisation is also the basis with which to predict future vulnerabilities in turn guiding the most appropriate deployment technologies. Thus, refreshing established practices and the scope of the training to support the decision making of users and operators. The foundation of the training provision is the use of Cyber-Ranges (CRs) and Test-Beds (TBs), platforms/tools that help inculcate a deeper understanding of the evolution of an attack and the methodology to deploy the most impactful countermeasures to arrest breaches. In this paper, an evaluation of documented CR and TB platforms is evaluated. CRs and TBs are segmented by type, technology, threat scenarios, applications and the scope of attainable training. To enrich the analysis of documented CR and TB research and cap the study, a taxonomy is developed to provide a broader comprehension of the future of CRs and TBs. The taxonomy elaborates on the CRs/TBs dimensions, as well as, highlighting a diminishing differentiation between application areas.

A Review of Cyber-Ranges and Test-Beds: Current and Future Trends

System logs, which describe a variety of events of software systems, are becoming increasingly popular for anomaly detection. However, for a large software system, current unsupervised learning-based methods are suffering from low accuracy due to the high diversity of logs, while the supervised learning methods are nearly infeasible to be used in practice because it is time-consuming and labor-intensive to obtain sufficient labels for different types of software systems. In this paper, we propose a novel framework, LogTransfer, which applies transfer learning to transfer the anomalous knowledge of one type of software system (source system) to another (target system). We represent every template using Glove, which considers both global word co-occurrence and local context information, to address the challenge that different types of software systems are different in log syntax while the semantics of logs should be reserved. We apply an LSTM network to extract the sequential patterns of logs, and propose a novel transfer learning method sharing fully connected networks between source and target systems, to minimize the impact of noises in anomalous log sequences. Extensive experiments have been performed on switch logs of different vendors collected from a top global cloud service provider. LogTransfer achieves an averaged 0.84 F1-score and outperforms the state-of-the-art supervised and unsupervised log-based anomaly detection methods, which are consistent with the experiments conducted on the public HDFS and Hadoop application datasets.

LogTransfer: Cross-System Log Anomaly Detection for Software Systems with Transfer Learning

System logs provide valuable information about the health status of IT systems and computer networks Therefore, log file monitoring has been identified as an important system and network management technique While many solutions have been developed for monitoring known log messages, the detection of previously unknown error conditions has remained a difficult problem In this paper, we present a novel data mining based framework for detecting anomalous log messages from syslog- based system log files We also describe the implementation and performance of the framework in a large organizational network

An unsupervised framework for detecting anomalous messages from syslog log files

The Internet Protocol Version 6 (IPv6) transition opens a wide scope for potential attack vectors. IPv6 transition mechanisms could allow the set-up of covert egress communication channels over an IPv4-only or dual-stack network, resulting in full compromise of a target network. Therefore effective tools are required for the execution of security operations for assessment of possible attack vectors related to IPv6 security.

Creating and Detecting IPv6 Transition Mechanism-Based Information Exfiltration Covert Channels

During the past two decades, event correlation has emerged as a prominent monitoring technique, and is essential for achieving better situational awareness. Since its introduction in 2001 by one of the authors of this paper, Simple Event Correlator (SEC) has become a widely used open source event correlation tool. During the last decade, a number of papers have been published that describe the use of SEC in various environments. However, recent SEC versions have introduced a number of novel features not discussed in existing works. This paper fills this gap and provides an up-to-date coverage of best practices for creating scalable SEC configurations.

/pdf/simple-event-correlator-best-practices-for-creating-scalable-4dnh7fboc5.pdf

Simple event correlator - Best practices for creating scalable configurations

Fuzzing is a critical part of secure software development life-cycle, for finding vulnerabilities, developing exploits, and reverse engineering. This relies on appropriate approaches, tools and frameworks. File and protocol fuzzing is well covered, multiple approaches and implementations exist. Unfortunately, assessed tools do not posses the required capabilities for working with protocols, where constructing bit groups are not byte aligned. In this paper, a systematic approach is proposed and tool prototype developed for the cyber red teaming purposes. In a case study, the developed Bbuzz tool is used to reverse engineer a proprietary NATO Link-1 network protocol allowing to inject rogue airplane tracks into air operations command and control system.

Bbuzz: A bit-aware fuzzing framework for network protocol systematic reverse engineering and analysis

Cyber Defense Exercises have received much attention in recent years, and are increasingly becoming the cornerstone for ensuring readiness in this new domain. Crossed Swords is an exercise directed at training Red Team members for responsive cyber defense. However, prior iterations have revealed the need for automated and transparent real-time feedback systems to help participants improve their techniques and understand technical challenges. Feedback was too slow and players did not understand the visibility of their actions. We developed a novel and modular open-source framework to address this problem, dubbed Frankenstack. We used this framework during Crossed Swords 2017 execution and evaluated its effectiveness by interviewing participants and conducting an online survey. Due to the novelty of Red Team-centric exercises, very little academic research exists on providing real-time feedback during such exercises. Thus, this paper serves as a first foray into a novel research field.

Bernhards Blumbergs

Papers

An unsupervised framework for detecting anomalous messages from syslog log files

Creating and Detecting IPv6 Transition Mechanism-Based Information Exfiltration Covert Channels

Simple event correlator - Best practices for creating scalable configurations

Bbuzz: A bit-aware fuzzing framework for network protocol systematic reverse engineering and analysis

Frankenstack: Toward real-time Red Team feedback