With increasing reliance on Internet of Things (IoT) devices and services, the capability to detect intrusions and malicious activities within IoT networks is critical for resilience of the network infrastructure. In this paper, we present a novel model for intrusion detection based on two-layer dimension reduction and two-tier classification module, designed to detect malicious activities such as User to Root (U2R) and Remote to Local (R2L) attacks. The proposed model is using component analysis and linear discriminate analysis of dimension reduction module to spate the high dimensional dataset to a lower one with lesser features. We then apply a two-tier classification module utilizing Naive Bayes and Certainty Factor version of K-Nearest Neighbor to identify suspicious behaviors. The experiment results using NSL-KDD dataset shows that our model outperforms previous models designed to detect U2R and R2L attacks.

/pdf/a-two-layer-dimension-reduction-and-two-tier-classification-4x8wrsx4te.pdf

A Two-Layer Dimension Reduction and Two-Tier Classification Model for Anomaly-Based Intrusion Detection in IoT Backbone Networks

One of the goals of smart environments is to improve the quality of human life in terms of comfort and efficiency. The Internet of Things (IoT) paradigm has recently evolved into a technology for building smart environments. Security and privacy are considered key issues in any real-world smart environment based on the IoT model. The security vulnerabilities in IoT-based systems create security threats that affect smart environment applications. Thus, there is a crucial need for intrusion detection systems (IDSs) designed for IoT environments to mitigate IoT-related security attacks that exploit some of these security vulnerabilities. Due to the limited computing and storage capabilities of IoT devices and the specific protocols used, conventional IDSs may not be an option for IoT environments. This article presents a comprehensive survey of the latest IDSs designed for the IoT model, with a focus on the corresponding methods, features, and mechanisms. This article also provides deep insight into the IoT architecture, emerging security vulnerabilities, and their relation to the layers of the IoT architecture. This work demonstrates that despite previous studies regarding the design and implementation of IDSs for the IoT paradigm, developing efficient, reliable and robust IDSs for IoT-based smart environments is still a crucial task. Key considerations for the development of such IDSs are introduced as a future outlook at the end of this survey.

/pdf/intrusion-detection-systems-for-iot-based-smart-environments-1btikhn15k.pdf

Intrusion detection systems for IoT-based smart environments: a survey

While machine learning and artificial intelligence have long been applied in networking research, the bulk of such works has focused on supervised learning. Recently, there has been a rising trend of employing unsupervised machine learning using unstructured raw network data to improve network performance and provide services, such as traffic engineering, anomaly detection, Internet traffic classification, and quality of service optimization. The growing interest in applying unsupervised learning techniques in networking stems from their great success in other fields, such as computer vision, natural language processing, speech recognition, and optimal control (e.g., for developing autonomous self-driving cars). In addition, unsupervised learning can unconstrain us from the need for labeled data and manual handcrafted feature engineering, thereby facilitating flexible, general, and automated methods of machine learning. The focus of this survey paper is to provide an overview of applications of unsupervised learning in the domain of networking. We provide a comprehensive survey highlighting recent advancements in unsupervised learning techniques, and describe their applications in various learning tasks, in the context of networking. We also provide a discussion on future directions and open research issues, while identifying potential pitfalls. While a few survey papers focusing on applications of machine learning in networking have previously been published, a survey of similar scope and breadth is missing in the literature. Through this timely review, we aim to advance the current state of knowledge, by carefully synthesizing insights from previous survey papers, while providing contemporary coverage of the recent advances and innovations.

/pdf/unsupervised-machine-learning-for-networking-techniques-1u9zi0s9j5.pdf

Unsupervised Machine Learning for Networking: Techniques, Applications and Research Challenges

Network intrusion detection plays a very important role in protecting computer network security. The abnormal traffic detection and analysis by extracting the statistical features of flow is the main analysis method in the field of network intrusion detection. However, these features need to be designed and extracted manually, which often loses the original information of the flow and leads to poor detection efficiency. In this paper, we do not manually design the features of the flow but directly extract the raw data information of the flow for analysis. In addition, we first proposed a new network intrusion detection model named the deep hierarchical network, which integrates the improved LeNet-5 and LSTM neural network structures, while learning the spatial and temporal features of flow. By designing a reasonable network cascading method, we can train our proposed hierarchical network at the same time instead of training two networks separately. In this paper, we use the CICIDS2017 dataset and the CTU dataset. The number and types of flow in these two datasets are large, and the attack types are relatively new. The experimental results show that the performance of the proposed hierarchical network model is significantly better than other network intrusion detection models, which can achieve the best detection accuracy. Finally, we also present an analysis method for traffic features which has an important contribution to abnormal traffic detection and gives the actual meanings of these important features.

/pdf/network-intrusion-detection-based-on-deep-hierarchical-3ofhs2kj2g.pdf

Network Intrusion Detection: Based on Deep Hierarchical Network and Original Flow Data

Advances in high-throughput sequencing on single-cell gene expressions [single-cell RNA sequencing (scRNA-seq)] have enabled transcriptome profiling on individual cells from complex samples. A common goal in scRNA-seq data analysis is to discover and characterise cell types, typically through clustering methods. The quality of the clustering therefore plays a critical role in biological discovery. While numerous clustering algorithms have been proposed for scRNA-seq data, fundamentally they all rely on a similarity metric for categorising individual cells. Although several studies have compared the performance of various clustering algorithms for scRNA-seq data, currently there is no benchmark of different similarity metrics and their influence on scRNA-seq data clustering. Here, we compared a panel of similarity metrics on clustering a collection of annotated scRNA-seq datasets. Within each dataset, a stratified subsampling procedure was applied and an array of evaluation measures was employed to assess the similarity metrics. This produced a highly reliable and reproducible consensus on their performance assessment. Overall, we found that correlation-based metrics (e.g. Pearson's correlation) outperformed distance-based metrics (e.g. Euclidean distance). To test if the use of correlation-based metrics can benefit the recently published clustering techniques for scRNA-seq data, we modified a state-of-the-art kernel-based clustering algorithm (SIMLR) using Pearson's correlation as a similarity measure and found significant performance improvement over Euclidean distance on scRNA-seq data clustering. These findings demonstrate the importance of similarity metrics in clustering scRNA-seq data and highlight Pearson's correlation as a favourable choice. Further comparison on different scRNA-seq library preparation protocols suggests that they may also affect clustering performance. Finally, the benchmarking framework is available at http://www.maths.usyd.edu.au/u/SMS/bioinformatics/software.html.

Impact of similarity metrics on single-cell RNA-seq data clustering.

Anomaly detection (AD) use within the network intrusion detection field of research, or network intrusion AD (NIAD), is dependent on the proper use of similarity and distance measures, but the measures used are often not documented in published research. As a result, while the body of NIAD research has grown extensively, knowledge of the utility of similarity and distance measures within the field has not grown correspondingly. NIAD research covers a myriad of domains and employs a diverse array of techniques from simple $k$ -means clustering through advanced multiagent distributed AD systems. This review presents an overview of the use of similarity and distance measures within NIAD research. The analysis provides a theoretical background in distance measures and a discussion of various types of distance measures and their uses. Exemplary uses of distance measures in published research are presented, as is the overall state of the distance measure rigor in the field. Finally, areas that require further focus on improving the distance measure rigor in the NIAD field are presented.

David J. Weller-Fahy

Papers

A Survey of Distance and Similarity Measures Used Within Network Intrusion Anomaly Detection