Showing papers on "Deep belief network published in 2020"

Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study

Abstract: In this paper, we present a survey of deep learning approaches for cybersecurity intrusion detection, the datasets used, and a comparative study. Specifically, we provide a review of intrusion detection systems based on deep learning approaches. The dataset plays an important role in intrusion detection, therefore we describe 35 well-known cyber datasets and provide a classification of these datasets into seven categories; namely, network traffic-based dataset, electrical network-based dataset, internet traffic-based dataset, virtual private network-based dataset, android apps-based dataset, IoT traffic-based dataset, and internet-connected devices-based dataset. We analyze seven deep learning models including recurrent neural networks, deep neural networks, restricted Boltzmann machines, deep belief networks, convolutional neural networks, deep Boltzmann machines, and deep autoencoders. For each model, we study the performance in two categories of classification (binary and multiclass) under two new real traffic datasets, namely, the CSE-CIC-IDS2018 dataset and the Bot-IoT dataset. In addition, we use the most important performance indicators, namely, accuracy, false alarm rate, and detection rate for evaluating the efficiency of several methods.

An Improved Quantum-Inspired Differential Evolution Algorithm for Deep Belief Network

Abstract: Deep belief network (DBN) is one of the most representative deep learning models. However, it has a disadvantage that the network structure and parameters are basically determined by experiences. In this article, an improved quantum-inspired differential evolution (MSIQDE), namely MSIQDE algorithm based on making use of the merits of the Mexh wavelet function, standard normal distribution, adaptive quantum state update, and quantum nongate mutation, is proposed to avoid premature convergence and improve the global search ability. Then, the MSIQDE with global optimization ability is used to optimize the parameters of the DBN to construct an optimal DBN model, which is further applied to propose a new fault classification, namely MSIQDE-DBN method. Finally, the vibration data of rolling bearings from the Case Western Reserve University and a real-world engineering application are carried out to verify the performance of the MSIQDE-DBN method. The experimental results show that the MSIQDE takes on better optimization performance, and the MSIQDE-DBN can obtain higher classification accuracy than the other comparison methods.

An Introductory Review of Deep Learning for Prediction Models With Big Data

Abstract: Deep learning models stand for a new learning paradigm in artificial intelligence (AI) and machine learning. Recent breakthrough results in image analysis and speech recognition have generated a massive interest in this field because also applications in many other domains providing big data seem possible. On a downside, the mathematical and computational methodology underlying deep learning models is very challenging, especially for interdisciplinary scientists. For this reason, we present in this paper an introductory review of deep learning approaches including Deep Feedforward Neural Networks (D-FFNN), Convolutional Neural Networks (CNNs), Deep Belief Networks (DBNs), Autoencoders (AEs), and Long Short-Term Memory (LSTM) networks. These models form the major core architectures of deep learning models currently used and should belong in any data scientist's toolbox. Importantly, those core architectural building blocks can be composed flexibly-in an almost Lego-like manner-to build new application-specific network architectures. Hence, a basic understanding of these network architectures is important to be prepared for future developments in AI.

A novel deep learning based fault diagnosis approach for chemical process with extended deep belief network.

Abstract: Deep learning networks have been recently utilized for fault detection and diagnosis (FDD) due to its effectiveness in handling industrial process data, which are often with high nonlinearities and strong correlations. However, the valuable information in the raw data may be filtered with the layer-wise feature compression in traditional deep networks. This cannot benefit for the subsequent fine-tuning phase of fault classification. To alleviate this problem, an extended deep belief network (EDBN) is proposed to fully exploit useful information in the raw data, in which raw data is combined with the hidden features as inputs to each extended restricted Boltzmann machine (ERBM) during the pre-training phase. Then, a dynamic EDBN-based fault classifier is constructed to take the dynamic characteristics of process data into consideration. Finally, to test the performance of the proposed method, it is applied to the Tennessee Eastman (TE) process for fault classification. By comparing EDBN and DBN under different network structures, the results show that EDBN has better feature extraction and fault classification performance than traditional DBN.

A Review on Deep Learning Methods for ECG Arrhythmia Classification

Abstract: Deep Learning (DL) has recently become a topic of study in different applications including healthcare, in which timely detection of anomalies on Electrocardiogram (ECG) can play a vital role in patient monitoring. This paper presents a comprehensive review study on the recent DL methods applied to the ECG signal for the classification purposes. This study considers various types of the DL methods such as Convolutional Neural Network (CNN), Deep Belief Network (DBN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU). From the 75 studies reported within 2017 and 2018, CNN is dominantly observed as the suitable technique for feature extraction, seen in 52% of the studies. DL methods showed high accuracy in correct classification of Atrial Fibrillation (AF) (100%), Supraventricular Ectopic Beats (SVEB) (99.8%), and Ventricular Ectopic Beats (VEB) (99.7%) using the GRU/LSTM, CNN, and LSTM, respectively.

Deep Semantic Feature Learning for Software Defect Prediction

Abstract: Software defect prediction, which predicts defective code regions, can assist developers in finding bugs and prioritizing their testing efforts. Traditional defect prediction features often fail to capture the semantic differences between different programs. This degrades the performance of the prediction models built on these traditional features. Thus, the capability to capture the semantics in programs is required to build accurate prediction models. To bridge the gap between semantics and defect prediction features, we propose leveraging a powerful representation-learning algorithm, deep learning, to learn the semantic representations of programs automatically from source code files and code changes. Specifically, we leverage a deep belief network (DBN) to automatically learn semantic features using token vectors extracted from the programs’ abstract syntax trees (AST) (for file-level defect prediction models) and source code changes (for change-level defect prediction models). We examine the effectiveness of our approach on two file-level defect prediction tasks (i.e., file-level within-project defect prediction and file-level cross-project defect prediction) and two change-level defect prediction tasks (i.e., change-level within-project defect prediction and change-level cross-project defect prediction). Our experimental results indicate that the DBN-based semantic features can significantly improve the examined defect prediction tasks. Specifically, the improvements of semantic features against existing traditional features (in F1) range from 2.1 to 41.9 percentage points for file-level within-project defect prediction, from 1.5 to 13.4 percentage points for file-level cross-project defect prediction, from 1.0 to 8.6 percentage points for change-level within-project defect prediction, and from 0.6 to 9.9 percentage points for change-level cross-project defect prediction.

Effective Attack Detection in Internet of Medical Things Smart Environment Using a Deep Belief Neural Network

Abstract: The Internet of Things (IoT) has lately developed into an innovation for developing smart environments. Security and privacy are viewed as main problems in any technology’s dependence on the IoT model. Privacy and security issues arise due to the different possible attacks caused by intruders. Thus, there is an essential need to develop an intrusion detection system for attack and anomaly identification in the IoT system. In this work, we have proposed a deep learning-based method Deep Belief Network (DBN) algorithm model for the intrusion detection system. Regarding the attacks and anomaly detection, the CICIDS 2017 dataset is utilized for the performance analysis of the present IDS model. The proposed method produced better results in all the parameters in relation to accuracy, recall, precision, F1-score, and detection rate. The proposed method has achieved 99.37% accuracy for normal class, 97.93% for Botnet class, 97.71% for Brute Force class, 96.67% for Dos/DDoS class, 96.37% for Infiltration class, 97.71% for Ports can class and 98.37% for Web attack, and these results were compared with various classifiers as shown in the results.

One-Dimensional Residual Convolutional Autoencoder Based Feature Learning for Gearbox Fault Diagnosis

Abstract: Vibration signals are generally utilized for machinery fault diagnosis to perform timely maintenance and then reduce losses. Thus, the feature extraction on one-dimensional vibration signals often determines accuracy of those fault diagnosis models. These typical deep neural networks (DNNs), e.g., convolutional neural networks (CNNs), perform well in feature learning and have been applied in machine fault diagnosis. However, the supervised learning of CNN often requires a large amount of labeled images and thus limits its wide applications. In this article, a new DNN, one-dimensional residual convolutional autoencoder (1-DRCAE), is proposed for learning features from vibration signals directly in an unsupervised-learning way. First, 1-D convolutional autoencoder is proposed in 1-DRCAE for feature extraction. Second, a deconvolution operation is developed as decoder of 1-DRCAE to reconstruct the filtered signals. Third, residual learning is employed in 1-DRCAE to perform feature learning on 1-D vibration signals. The results show that 1-DRCAE has good signal denoising and feature extraction performance on vibration signals. It performs better on feature extraction than the typical DNNs, e.g., deep belief network, stacked autoencoders, and 1-D CNN.

An Enhanced Intelligent Diagnosis Method Based on Multi-Sensor Image Fusion via Improved Deep Learning Network

Abstract: An enhanced intelligent diagnosis method for rotary equipment is proposed based on multi-sensor data-fusion and an improved deep convolutional neural network (CNN) models An improved CNN based on LeNet-5 is constructed which can enhance the features of the samples by stacking bottleneck layers without changing the size of the samples A new conversion approaches are also proposed for converting multi-sensor vibration signals into color images, and it can refine features and enlarge the differences between different types of fault signals by the fused images transformed in red–green–blue (RGB) color space In the last stage of network learning, visual clustering is realized with t-distributed stochastic neighbor embedding (t-SNE) to evaluate the performance of the network To verify the effectiveness of the proposed method, examples in practice such as the diagnosis for the wind power test rigs and industrial fan system are provided with the prediction accuracies of 9989% and 9977%, respectively In addition, the efficiency of other comparative baseline approaches such as the deep belief network and support vector machine (SVM) is evaluated In conclusion, the proposed intelligent diagnosis method based on multi-sensor data-fusion and CNN shows higher prediction accuracy and more obvious visualization clustering effects

Non-iterative and Fast Deep Learning: Multilayer Extreme Learning Machines

Abstract: In the past decade, deep learning techniques have powered many aspects of our daily life, and drawn ever-increasing research interests. However, conventional deep learning approaches, such as deep belief network (DBN), restricted Boltzmann machine (RBM), and convolutional neural network (CNN), suffer from time-consuming training process due to fine-tuning of a large number of parameters and the complicated hierarchical structure. Furthermore, the above complication makes it difficult to theoretically analyze and prove the universal approximation of those conventional deep learning approaches. In order to tackle the issues, multilayer extreme learning machines (ML-ELM) were proposed, which accelerate the development of deep learning. Compared with conventional deep learning, ML-ELMs are non-iterative and fast due to the random feature mapping mechanism. In this paper, we perform a thorough review on the development of ML-ELMs, including stacked ELM autoencoder (ELM-AE), residual ELM, and local receptive field based ELM (ELM-LRF), as well as address their applications. In addition, we also discuss the connection between random neural networks and conventional deep learning.

Deep Reinforcement Learning for Dynamic Uplink/Downlink Resource Allocation in High Mobility 5G HetNet

Abstract: Recently, the 5G is widely deployed for supporting communications of high mobility nodes including train, vehicular and unmanned aerial vehicles (UAVs) largely emerged as the main components for constructing the wireless heterogeneous network (HetNet). To further improve the radio utilization, the Time Division Duplex (TDD) is considered to be the potential full-duplex communication technology in the high mobility 5G network. However, the high mobility of users leads to the high dynamic network traffic and unpredicted link state change. A new method to predict the dynamic traffic and channel condition and schedule the TDD configuration in real-time is essential for the high mobility environment. In this paper, we investigate the channel model in the high mobility and heterogeneous network and proposed a novel deep reinforcement learning based intelligent TDD configuration algorithm to dynamically allocate radio resources in an online manner. In the proposal, the deep neural network is employed to extract the features of the complex network information, and the dynamic Q-value iteration based reinforcement learning with experience replay memory mechanism is proposed to adaptively change TDD Up/Down-link ratio by evaluated rewards. The simulation results show that the proposal achieves significant network performance improvement in terms of both network throughput and packet loss rate, comparing with conventional TDD resource allocation algorithms.

Prevalence and Diagnosis of Neurological Disorders Using Different Deep Learning Techniques: A Meta-Analysis.

Abstract: This paper dispenses an exhaustive review on deep learning techniques used in the prognosis of eight different neuropsychiatric and neurological disorders such as stroke, alzheimer, parkinson’s, epilepsy, autism, migraine, cerebral palsy, and multiple sclerosis. These diseases are critical, life-threatening and in most of the cases may lead to other precarious human disorders. Deep learning techniques are emerging soft computing technique which has been lucratively used to unravel different real-life problems such as pattern recognition (Face, Emotion, and Speech), traffic management, drug discovery, disease diagnosis, and network intrusion detection. This study confers the discipline, frameworks, and methodologies used by different deep learning techniques to diagnose different human neurological disorders. Here, one hundred and thirty-six different articles related to neurological and neuropsychiatric disorders diagnosed using different deep learning techniques are studied. The morbidity and mortality rate of major neuropsychiatric and neurological disorders has also been delineated. The performance and publication trend of different deep learning techniques employed in the investigation of these diseases has been examined and analyzed. Different performance metrics like accuracy, specificity, and sensitivity have also been examined. The research implication, challenges and the future directions related to the study have also been highlighted. Eventually, the research breaches are identified and it is witnessed that there is more scope in the diagnosis of migraine, cerebral palsy and stroke using different deep learning models. Likewise, there is a potential opportunity to use and explore the performance of Restricted Boltzmann Machine, Deep Boltzmann Machine and Deep Belief Network for diagnosis of different human neuropsychiatric and neurological disorders.

Facial Sentiment Analysis Using AI Techniques: State-of-the-Art, Taxonomies, and Challenges

Abstract: With the advancements in machine and deep learning algorithms, the envision of various critical real-life applications in computer vision becomes possible. One of the applications is facial sentiment analysis. Deep learning has made facial expression recognition the most trending research fields in computer vision area. Recently, deep learning-based FER models have suffered from various technological issues like under-fitting or over-fitting. It is due to either insufficient training and expression data. Motivated from the above facts, this paper presents a systematic and comprehensive survey on current state-of-art Artificial Intelligence techniques (datasets and algorithms) that provide a solution to the aforementioned issues. It also presents a taxonomy of existing facial sentiment analysis strategies in brief. Then, this paper reviews the existing novel machine and deep learning networks proposed by researchers that are specifically designed for facial expression recognition based on static images and present their merits and demerits and summarized their approach. Finally, this paper also presents the open issues and research challenges for the design of a robust facial expression recognition system.

Multiscale cascading deep belief network for fault identification of rotating machinery under various working conditions

Abstract: Deep learning is characterized by strong self-learning and fault classification ability without manually feature extraction stage of traditional algorithms. Deep belief network (DBN) is one of the most classic models of deep learning. However, traditional DBN is mainly restricted to learn automatically single scale features from raw vibration signal while identify the fault type, which implies some important information inherent in other scales of vibration data are neglected, thus causing easily unsatisfactory diagnosis result. To alleviate the problem, this paper presents a novel architecture named multiscale cascading deep belief network (MCDBN) for automatic fault identification of rotating machinery, which is aimed at learning the broader feature representation and improving the recognition precision. Firstly, a sliding window with data overlap is adopted to split the collected raw vibration signal to a group of equal-sized sub-signal, and then the improved multiscale coarse-grained procedure of each sub-signal is conducted to obtain the coarse-grained time series at different scales. Meanwhile, Fourier spectrum at different scales is calculated to capture multiscale characteristics. Finally, multiple DBN architecture with three hidden layers are designed to learn high-level feature representation directly from multiscale characteristics in a parallel manner and accomplish fault identification automatically through cascading way and softmax classifier without artificial expertise. Results of two experimental cases with respect to mechanical fault identification under different working conditions have well indicated that the proposed method is provided with preferable diagnostic performance compared with standard DBN and traditional multiscale feature extractors.

A Novel Application of Deep Belief Networks in Learning Partial Discharge Patterns for Classifying Corona, Surface, and Internal Discharges

Abstract: This paper introduces a new application of deep belief network (DBN) as an emerging artificial neural network for recognizing and classifying different partial discharge (PD) patterns. Phase resolved PD (PRPD) technique with different window intervals is used to manipulate three PD types, including corona, surface, and internal discharges measured in a high-voltage lab. Four approaches are proposed for extracting features from the raw measured data. In the first approach, the DBN is used as both a feature extractor and a PD classifier. The other three approaches extract discriminatory features using statistical and vector-norm-based operators to train a DBN classifier. The impact of the various phase windows through the PRPD method on the performance of the trained classifier is evaluated to obtain the best window interval. It is shown that the deep architectures are capable of learning important distinguishable features from PD data without any data preprocessing. This eliminates time-consuming feature extraction processes that produce the handcrafted features. Based on a comparison analysis, when the input data are corrupted by noise levels or no feature extraction technique is used to preprocess the data, the proposed approach outperforms other techniques, such as artificial neural networks, fuzzy logic classifiers, and support vector machines.

An intrusion detection approach based on improved deep belief network

Abstract: In today’s interconnected society, cyberattacks have become more frequent and sophisticated, and existing intrusion detection systems may not be adequate in the complex cyberthreat landscape. For instance, existing intrusion detection systems may have overfitting, low classification accuracy, and high false positive rate (FPR) when faced with significantly large volume and variety of network data. An intrusion detection approach based on improved deep belief network (DBN) is proposed in this paper to mitigate the above problems, where the dataset is processed by probabilistic mass function (PMF) encoding and Min-Max normalization method to simplify the data preprocessing. Furthermore, a combined sparsity penalty term based on Kullback-Leibler (KL) divergence and non-mean Gaussian distribution is introduced in the likelihood function of the unsupervised training phase of DBN, and sparse constraints retrieve the sparse distribution of the dataset, thus avoiding the problem of feature homogeneity and overfitting. Finally, simulation experiments are performed on the NSL-KDD and UNSW-NB15 public datasets. The proposed method achieves 96.17% and 86.49% accuracy, respectively. Experimental results show that compared with the state-of-the-art methods, the proposed method achieves significant improvement in classification accuracy and FPR.

Two-Terminal Fault Location Method of Distribution Network Based on Adaptive Convolution Neural Network

Abstract: When a single-phase ground fault occurs in a distribution network, it is generally allowed to operate with faults for one to two hours, which may lead to further development of the fault and even threaten the safe operation of the power system. Therefore, when a small current system has a ground fault, it must be quickly diagnosed to shorten the time of operation with fault. In this paper, an adaptive convolutional neural network (ACNN)-based fault line selection method is proposed for a distribution network. This method improves the feature extraction ability of the network by improving the pooling model. Compared with deep belief network (DBN), it can improve the accuracy of fault classification by 7.86% and reduce the training time by 42.7%. On this basis, the secondary fault location is identified using the principle of two-terminal fault location. In this research, fault data obtained by Simulink simulation is used as training set, and ACNN model is built based on TensorFlow framework. The analysis of results proves that the model has a high fault recognition rate and fast convergence speed. It can be used as an auxiliary hand for fault diagnosis in distribution networks.

A real-time and ubiquitous network attack detection based on deep belief network and support vector machine

Abstract: In recent years, network traffic data have become larger and more complex, leading to higher possibilities of network intrusion. Traditional intrusion detection methods face difficulty in processing high-speed network data and cannot detect currently unknown attacks. Therefore, this paper proposes a network attack detection method combining a flow calculation and deep learning. The method consists of two parts: a real-time detection algorithm based on flow calculations and frequent patterns and a classification algorithm based on the deep belief network and support vector machine ( DBN-SVM ) . Sliding window ( SW ) stream data processing enables real-time detection, and the DBN-SVM algorithm can improve classification accuracy. Finally, to verify the proposed method, a system is implemented. Based on the CICIDS2017 open source data set, a series of comparative experiments are conducted. The method ʼ s real-time detection efficiency is higher than that of traditional machine learning algorithms. The attack classification accuracy is 0.7 percentage points higher than that of a DBN, which is 2 percentage points higher than that of the integrated algorithm boosting and bagging methods. Hence, it is suitable for the real-time detection of high-speed network intrusions.

Hybrid Ensemble Deep Learning for Deterministic and Probabilistic Low-Voltage Load Forecasting

Abstract: Accurate and reliable low-voltage load forecasting is critical to optimal operation and control of distribution network and smart grid. However, compared to traditional regional load forecasting at high-voltage level, it faces tough challenges due to the inherent high uncertainty of the low-capacity load and distributed renewable energy integrated in the demand side. This paper proposes a novel hybrid ensemble deep learning (HEDL) approach for deterministic and probabilistic low-voltage load forecasting. The deep belief network (DBN) is applied to low-voltage load point prediction with the strong ability of approximating nonlinear mapping. A series of ensemble learning methods including bagging and boosting variants are introduced to improve the regression ability of DBN. In addition, the differencing transformation technique is utilized to ensure the stationarity of load time series for the application bagging and boosting methods. On the basis of the integrated thought of ensemble learning, a new hybrid ensemble algorithm is developed via integrating multiple separate ensemble methods. Considering the diversity in various ensemble algorithms, an effective K nearest neighbor classification method is utilized to adaptively determine the weights of sub-models. Furthermore, HEDL based probabilistic forecasting is proposed by taking advantage of the inherent resample idea in bagging and boosting. The effectiveness of the HEDL method for both deterministic and probabilistic forecasting has been systematically verified based on realistic load data from East China and Australia, indicating its promising prospective for practical applications in distribution networks.