The rise of machine learning for detection and classification of malware: Research developments, trends and challenges

Threats that have been primarily targeting nation states and their associated entities have expanded the target zone to include the private and corporate sectors. This class of threats, well known as advanced persistent threats (APTs), are those that every nation and well-established organization fears and wants to protect itself against. While nation-sponsored APT attacks will always be marked by their sophistication, APT attacks that have become prominent in corporate sectors do not make it any less challenging for the organizations. The rate at which the attack tools and techniques are evolving is making any existing security measures inadequate. As defenders strive to secure every endpoint and every link within their networks, attackers are finding new ways to penetrate into their target systems. With each day bringing new forms of malware, having new signatures and behavior that is close to normal, a single threat detection system would not suffice. While it requires time and patience to perform APT, solutions that adapt to the changing behavior of APT attacker(s) are required. Several works have been published on detecting an APT attack at one or two of its stages, but very limited research exists in detecting APT as a whole from reconnaissance to cleanup, as such a solution demands complex correlation and fine-grained behavior analysis of users and systems within and across networks. Through this survey paper, we intend to bring all those methods and techniques that could be used to detect different stages of APT attacks, learning methods that need to be applied and where to make your threat detection framework smart and undecipherable for those adapting APT attackers. We also present different case studies of APT attacks, different monitoring methods, and mitigation methods to be employed for fine-grained control of security of a networked system. We conclude this paper with different challenges in defending against APT and opportunities for further research, ending with a note on what we learned during our writing of this paper.

A Survey on Advanced Persistent Threats: Techniques, Solutions, Challenges, and Research Opportunities

Industrial Internet of Things (IIoT) is considered as one of the most promising revolutionary technologies to prompt smart manufacturing and increase productivity. With manufacturing being more complicated and sophisticated, an entire manufacturing process usually involves several different administrative IoT domains (e.g., factories). Devices from different domains collaborate on the same task, which raises great security and privacy concerns about device-to-device communications. Existing authentication approaches may result in heavy key management overhead or rely on a trusted third party. Thus, security and privacy issues during communication remain unsolved but imperative. In this paper, we present an efficient blockchain-assisted secure device authentication mechanism $\textsf{BASA}$ for cross-domain IIoT. Specifically, consortium blockchain is introduced to construct trust among different domains. Identity-based signature (IBS) is exploited during the authentication process. To preserve the privacy of devices, we design an identity management mechanism, which can realize that devices being authenticated remain anonymous. Besides, session keys between two parties are negotiated, which can secure the subsequent communications. Extensive experiments have been conducted to show the effectiveness and efficiency of the proposed mechanism.

Blockchain-Assisted Secure Device Authentication for Cross-Domain Industrial IoT

Advanced Persistent Threat (APT) is a serious threat against sensitive information. Current detection approaches are time-consuming since they detect APT attack by in-depth analysis of massive amounts of data after data breaches. Specifically, APT attackers make use of DNS to locate their command and control (C&C) servers and victims’ machines. In this paper, we propose an efficient approach to detect APT malware C&C domain with high accuracy by analyzing DNS logs. We first extract 15 features from DNS logs of mobile devices. According to Alexa ranking and the VirusTotal’s judgement result, we give each domain a score. Then, we select the most normal domains by the score metric. Finally, we utilize our anomaly detection algorithm, called Global Abnormal Forest (GAF), to identify malware C&C domains. We conduct a performance analysis to demonstrate that our approach is more efficient than other existing works in terms of calculation efficiency and recognition accuracy. Compared with Local Outlier Factor (LOF), -Nearest Neighbor (KNN), and Isolation Forest (iForest), our approach obtains more than 99%  and for the detection of C&C domains. Our approach not only can reduce data volume that needs to be recorded and analyzed but also can be applicable to unsupervised learning.

/pdf/identifying-apt-malware-domain-based-on-mobile-dns-logging-fe7kuxruk0.pdf

Identifying APT Malware Domain Based on Mobile DNS Logging

Network attack behavior detection using deep learning is an important research topic in the field of network security. Currently, there are still many challenges in detecting multi-class imbalanced abnormal traffic data. This paper proposed a new intrusion detection network based on deep learning, named parallel cross convolutional neural network (PCCN), to improve the detection performance of imbalanced abnormal flows. By fusing the flow features learned from the two branch convolutional neural networks (CNN), PCCN can better learn the flow features with fewer samples, to improve the detection results of the imbalanced abnormal flows. We proposed an improved feature extraction method of the original flow to extract multi-class flow features at the same time. The proposed algorithm not only reduces the number of useless elements for network learning, but also accelerates network convergence. In addition, we proposed four improved versions of the PCCN network structure to meet the real-time requirements of network intrusion detection in the current big data computing. These networks can achieve almost the same detection results as the PCCN, but greatly reduce the detection time of data. Through the analysis of high-order evaluation metrics, the proposed PCCN algorithm is significantly better than the traditional machine learning algorithms. Compared with the current hierarchical network model, PCCN can also achieve better performance in term of overall accuracy.

/pdf/pccn-parallel-cross-convolutional-neural-network-for-3c49ssynzw.pdf

PCCN: Parallel Cross Convolutional Neural Network for Abnormal Network Traffic Flows Detection in Multi-Class Imbalanced Network Traffic Flows

Advanced persistent threat (APT) is a serious threat to the Internet. With the aid of APT malware, attackers can remotely control infected machines and steal sensitive information. DNS is popular for malware to locate command and control (C&C) servers. In this paper, we propose a novel system placed at the network egress point that aims to efficiently and effectively detect APT malware infections based on malicious DNS and traffic analysis. The system uses malicious DNS analysis techniques to detect suspicious APT malware C&C domains, and then analyzes the traffic of the corresponding suspicious IP using the signature-based and anomaly based detection technology. We extracted 14 features based on big data to characterize different properties of malware-related DNS and the ways that they are queried, and we also defined network traffic features that can identify the traffic of compromised clients that have remotely been controlled. We built a reputation engine to compute a reputation score for an IP address using these features vector together. Our experiment was performed at a large local institute network for two months, and all the features were studied with big data, which includes $\sim 400$ million DNS queries. Our security approach cannot only substantially reduce the volume of network traffic that needs to be recorded and analyzed but also improve the sustainability of the system.

G. Zhao

Papers

Detecting APT Malware Infections Based on Malicious DNS and Traffic Analysis