The Internet of Things (IoT) appliances often expose sensitive data, either directly or indirectly. They may, for instance, tell whether you are at home right now or what your long or short-term habits are. Therefore, it is crucial to protect such devices against adversaries and has in place an early warning system which indicates compromised devices in a quick and efficient manner. In this paper, we propose time window embedding solutions that efficiently process a massive amount of data and have a low-memory-footprint at the same time. On top of the proposed embedding vectors, we use the core anomaly detection unit. It is a classifier that is based on the transformer’s encoder component followed by a feed-forward neural network. We have compared the proposed method with other classical machine-learning algorithms. Therefore, in the paper, we formally evaluate various machine-learning schemes and discuss their effectiveness in the IoT-related context. Our proposal is supported by detailed experiments that have been conducted on the recently published Aposemat IoT-23 dataset.

/pdf/a-new-method-of-hybrid-time-window-embedding-with-xwhj6yf5eb.pdf

A new method of hybrid time window embedding with transformer-based traffic data classification in IoT-networked environment

/pdf/detection-of-cyberattacks-traces-in-iot-data-5ut1fuw0q1.pdf

Detection of Cyberattacks Traces in IoT Data

/pdf/evaluating-standard-feature-sets-towards-increased-3w413w2c.pdf

Evaluating Standard Feature Sets Towards Increased Generalisability and Explainability of ML-Based Network Intrusion Detection

Network flow-based cyber anomaly detection is a difficult and complex task. Although several approaches to tackling this problem have been suggested, many research topics remain open. One of these concerns the problem of model transferability. There is a limited number of papers which tackle transfer learning in the context of flow-based network anomaly detection, and the proposed approaches are mostly evaluated on outdated datasets. The majority of solutions employ various sophisticated approaches, where different architectures of shallow and deep machine learning are leveraged. Analysis and experimentation show that different solutions achieve remarkable performance in a single domain, but transferring the performance to another domain is tedious and results in serious deterioration in prediction quality. In this paper, an innovative approach is proposed which adapts sketchy data structures to extract generic and universal features and leverages the principles of domain adaptation to improve classification quality in zero- and few-shot scenarios. The proposed approach achieves an F1 score of 0.99 compared to an F1 score of 0.97 achieved by the best-performing related methods.

https://www.mdpi.com/2076-3417/12/19/9636/pdf?version=1664184864

Towards Zero-Shot Flow-Based Cyber-Security Anomaly Detection Framework

The number of security breaches in the cyberspace is on the rise. This threat is met with intensive work in the intrusion detection research community. To keep the defensive mechanisms up to date and relevant, realistic network traffic datasets are needed. The use of flow-based data for machine-learning-based network intrusion detection is a promising direction for intrusion detection systems. However, many contemporary benchmark datasets do not contain features that are usable in the wild. The main contribution of this work is to cover the research gap related to identifying and investigating valuable features in the NetFlow schema that allow for effective, machine-learning-based network intrusion detection in the real world. To achieve this goal, several feature selection techniques have been applied on five flow-based network intrusion detection datasets, establishing an informative flow-based feature set. The authors’ experience with the deployment of this kind of system shows that to close the research-to-market gap, and to perform actual real-world application of machine-learning-based intrusion detection, a set of labeled data from the end-user has to be collected. This research aims at establishing the appropriate, minimal amount of data that is sufficient to effectively train machine learning algorithms in intrusion detection. The results show that a set of 10 features and a small amount of data is enough for the final model to perform very well.

How to Effectively Collect and Process Network Data for Intrusion Detection

Cybersecurity is an arms race, with both the security and the adversaries attempting to outsmart one another, coming up with new attacks, new ways to defend against those attacks, and again with new ways to circumvent those defences. This situation creates a constant need for novel, realistic cybersecurity datasets. This paper introduces the effects of using machine-learning-based intrusion detection methods in network traffic coming from a real-life architecture. The main contribution of this work is a dataset coming from a real-world, academic network. Real-life traffic was collected and, after performing a series of attacks, a dataset was assembled. The dataset contains 44 network features and an unbalanced distribution of classes. In this work, the capability of the dataset for formulating machine-learning-based models was experimentally evaluated. To investigate the stability of the obtained models, cross-validation was performed, and an array of detection metrics were reported. The gathered dataset is part of an effort to bring security against novel cyberthreats and was completed in the SIMARGL project.

https://www.mdpi.com/1424-8220/21/13/4319/pdf

The Proposition and Evaluation of the RoEduNet-SIMARGL2021 Network Intrusion Detection Dataset.

In this paper, the performance of stream processing and accuracy in the prediction of suspicious flows in simulated network traffic is investigated. In addition, concepts of an engine that integrates with novel solutions like the Elastic-search database and Apache Kafka that allows easy definition of streams and implementation of any machine learning algorithm are presented.

Real-time stream processing tool for detecting suspicious network patterns using machine learning

Handling the data imbalance problem is one of the crucial steps in a machine learning pipeline. The research community is well aware of the effects of data imbalance on machine learning algorithms. At the same time, there is a rising need for explainability of AI, especially in difficult, high-stake domains like network intrusion detection. In this paper, the effects of data balancing procedures on two explainability procedures implemented to explain a neural network used for network intrusion detection are evaluated. The discrepancies between the two methods are highlighted and important conclusions are drawn.

The Proposition of Balanced and Explainable Surrogate Method for Network Intrusion Detection in Streamed Real Difficult Data.

There is a profuse abundance of network security incidents around the world every day. Increasingly, services and data stored on servers fall victim to sophisticated techniques that cause all sorts of damage. Hackers invent new ways to bypass security measures and modify the existing viruses in order to deceive defense systems. Therefore, in response to these illegal procedures, new ways to defend against them are being developed. In this paper, a method for anomaly detection based on machine learning technique is presented and a near real-time processing system architecture is proposed. The main contribution is a test-run of ML algorithms on real-world data coming from a world-class telecom operator. This work investigates the effectiveness of detecting malicious behaviour in network packets using several machine learning techniques. The results achieved are expressed with a set of metrics. For better clarity on the classifier performance, 10-fold cross-validation was used.

Mikołaj Komisarek

Papers

The Proposition and Evaluation of the RoEduNet-SIMARGL2021 Network Intrusion Detection Dataset.

Real-time stream processing tool for detecting suspicious network patterns using machine learning

How to Effectively Collect and Process Network Data for Intrusion Detection

The Proposition of Balanced and Explainable Surrogate Method for Network Intrusion Detection in Streamed Real Difficult Data.

Network Intrusion Detection in the Wild - the Orange use case in the SIMARGL project