scispace - formally typeset
Search or ask a question

Showing papers on "Deep belief network published in 2017"


Proceedings ArticleDOI
04 Aug 2017
TL;DR: Novel extensions to deep autoencoders are demonstrated which not only maintain a deep autenkocoders' ability to discover high quality, non-linear features but can also eliminate outliers and noise without access to any clean training data.
Abstract: Deep autoencoders, and other deep neural networks, have demonstrated their effectiveness in discovering non-linear features across many problem domains. However, in many real-world problems, large outliers and pervasive noise are commonplace, and one may not have access to clean training data as required by standard deep denoising autoencoders. Herein, we demonstrate novel extensions to deep autoencoders which not only maintain a deep autoencoders' ability to discover high quality, non-linear features but can also eliminate outliers and noise without access to any clean training data. Our model is inspired by Robust Principal Component Analysis, and we split the input data X into two parts, $X = L_{D} + S$, where $L_{D}$ can be effectively reconstructed by a deep autoencoder and $S$ contains the outliers and noise in the original data X. Since such splitting increases the robustness of standard deep autoencoders, we name our model a "Robust Deep Autoencoder (RDA)". Further, we present generalizations of our results to grouped sparsity norms which allow one to distinguish random anomalies from other types of structured corruptions, such as a collection of features being corrupted across many instances or a collection of instances having more corruptions than their fellows. Such "Group Robust Deep Autoencoders (GRDA)" give rise to novel anomaly detection approaches whose superior performance we demonstrate on a selection of benchmark problems.

1,030 citations


Journal ArticleDOI
TL;DR: Experimental results demonstrated that the proposed SAE-DBN approach can effectively identify the machine running conditions and significantly outperform other fusion methods.
Abstract: To assess health conditions of rotating machinery efficiently, multiple accelerometers are mounted on different locations to acquire a variety of possible faults signals. The statistical features are extracted from these signals to identify the running status of a machine. However, the acquired vibration signals are different due to sensor’s arrangement and environmental interference, which may lead to different diagnostic results. In order to improve the fault diagnosis reliability, a new multisensor data fusion technique is proposed. First, time-domain and frequency-domain features are extracted from the different sensor signals, and then these features are input into multiple two-layer sparse autoencoder (SAE) neural networks for feature fusion. Finally, fused feature vectors can be regarded as the machine health indicators, and be used to train deep belief network (DBN) for further classification. To verify the effectiveness of the proposed SAE-DBN scheme, the bearing fault experiments were conducted on a bearing test platform, and the vibration data sets under different running speeds were collected for algorithm validation. For comparison, different feature fusion methods were also applied to multisensor fusion in the experiments. Experimental results demonstrated that the proposed approach can effectively identify the machine running conditions and significantly outperform other fusion methods.

632 citations


Journal ArticleDOI
TL;DR: A multiobjective deep belief networks ensemble (MODBNE) method that employs a multiobjectives evolutionary algorithm integrated with the traditional DBN training technique to evolve multiple DBNs simultaneously subject to accuracy and diversity as two conflicting objectives is proposed.
Abstract: In numerous industrial applications where safety, efficiency, and reliability are among primary concerns, condition-based maintenance (CBM) is often the most effective and reliable maintenance policy. Prognostics, as one of the key enablers of CBM, involves the core task of estimating the remaining useful life (RUL) of the system. Neural networks-based approaches have produced promising results on RUL estimation, although their performances are influenced by handcrafted features and manually specified parameters. In this paper, we propose a multiobjective deep belief networks ensemble (MODBNE) method. MODBNE employs a multiobjective evolutionary algorithm integrated with the traditional DBN training technique to evolve multiple DBNs simultaneously subject to accuracy and diversity as two conflicting objectives. The eventually evolved DBNs are combined to establish an ensemble model used for RUL estimation, where combination weights are optimized via a single-objective differential evolution algorithm using a task-oriented objective function. We evaluate the proposed method on several prognostic benchmarking data sets and also compare it with some existing approaches. Experimental results demonstrate the superiority of our proposed method.

569 citations


Journal ArticleDOI
TL;DR: This work introduces a recurrent deep neural network for real-time financial signal representation and trading and proposes a task-aware backpropagation through time method to cope with the gradient vanishing issue in deep training.
Abstract: Can we train the computer to beat experienced traders for financial assert trading? In this paper, we try to address this challenge by introducing a recurrent deep neural network (NN) for real-time financial signal representation and trading. Our model is inspired by two biological-related learning concepts of deep learning (DL) and reinforcement learning (RL). In the framework, the DL part automatically senses the dynamic market condition for informative feature learning. Then, the RL module interacts with deep representations and makes trading decisions to accumulate the ultimate rewards in an unknown environment. The learning system is implemented in a complex NN that exhibits both the deep and recurrent structures. Hence, we propose a task-aware backpropagation through time method to cope with the gradient vanishing issue in deep training. The robustness of the neural system is verified on both the stock and the commodity future markets under broad testing conditions.

522 citations


Proceedings ArticleDOI
19 Aug 2017
TL;DR: Variational Deep Embedding (VaDE) is proposed, a novel unsupervised generative clustering approach within the framework of Variational Auto-Encoder (VAE), which shows its capability of generating highly realistic samples for any specified cluster, without using supervised information during training.
Abstract: Clustering is among the most fundamental tasks in machine learning and artificial intelligence. In this paper, we propose Variational Deep Embedding (VaDE), a novel unsupervised generative clustering approach within the framework of Variational Auto-Encoder (VAE). Specifically, VaDE models the data generative procedure with a Gaussian Mixture Model (GMM) and a deep neural network (DNN): 1) the GMM picks a cluster; 2) from which a latent embedding is generated; 3) then the DNN decodes the latent embedding into an observable. Inference in VaDE is done in a variational way: a different DNN is used to encode observables to latent embeddings, so that the evidence lower bound (ELBO) can be optimized using the Stochastic Gradient Variational Bayes (SGVB) estimator and the reparameterization trick. Quantitative comparisons with strong baselines are included in this paper, and experimental results show that VaDE significantly outperforms the state-of-the-art clustering methods on 5 benchmarks from various modalities. Moreover, by VaDE's generative nature, we show its capability of generating highly realistic samples for any specified cluster, without using supervised information during training.

356 citations


Journal ArticleDOI
TL;DR: An unsupervised deep network, called the stacked convolutional denoising auto-encoders, which can map images to hierarchical representations without any label information is proposed, which demonstrates superior classification performance to state-of-the-art un supervised networks.
Abstract: Deep networks have achieved excellent performance in learning representation from visual data. However, the supervised deep models like convolutional neural network require large quantities of labeled data, which are very expensive to obtain. To solve this problem, this paper proposes an unsupervised deep network, called the stacked convolutional denoising auto-encoders, which can map images to hierarchical representations without any label information. The network, optimized by layer-wise training, is constructed by stacking layers of denoising auto-encoders in a convolutional way. In each layer, high dimensional feature maps are generated by convolving features of the lower layer with kernels learned by a denoising auto-encoder. The auto-encoder is trained on patches extracted from feature maps in the lower layer to learn robust feature detectors. To better train the large network, a layer-wise whitening technique is introduced into the model. Before each convolutional layer, a whitening layer is embedded to sphere the input data. By layers of mapping, raw images are transformed into high-level feature representations which would boost the performance of the subsequent support vector machine classifier. The proposed algorithm is evaluated by extensive experimentations and demonstrates superior classification performance to state-of-the-art unsupervised networks.

350 citations


Journal ArticleDOI
01 Jun 2017
TL;DR: A novel deep learning scheme based on restricted Boltzmann machine (RBM) is proposed, which shows that the performance improvement of FDBN over other selected state-of-the-art methods is statistically significant.
Abstract: Motor imagery classification is an important topic in brain–computer interface (BCI) research that enables the recognition of a subject’s intension to, e.g., implement prosthesis control. The brain dynamics of motor imagery are usually measured by electroencephalography (EEG) as nonstationary time series of low signal-to-noise ratio. Although a variety of methods have been previously developed to learn EEG signal features, the deep learning idea has rarely been explored to generate new representation of EEG features and achieve further performance improvement for motor imagery classification. In this study, a novel deep learning scheme based on restricted Boltzmann machine (RBM) is proposed. Specifically, frequency domain representations of EEG signals obtained via fast Fourier transform (FFT) and wavelet package decomposition (WPD) are obtained to train three RBMs. These RBMs are then stacked up with an extra output layer to form a four-layer neural network, which is named the frequential deep belief network (FDBN). The output layer employs the softmax regression to accomplish the classification task. Also, the conjugate gradient method and backpropagation are used to fine tune the FDBN. Extensive and systematic experiments have been performed on public benchmark datasets, and the results show that the performance improvement of FDBN over other selected state-of-the-art methods is statistically significant. Also, several findings that may be of significant interest to the BCI community are presented in this article.

347 citations


Journal ArticleDOI
01 May 2017
TL;DR: An ensemble deep learning method has been proposed for load demand forecasting that composes of Empirical Mode Decomposition and Deep Belief Network and results demonstrated attractiveness of the proposed method compared with nine forecasting methods.
Abstract: Graphical abstractDisplay Omitted HighlightsAn ensemble deep learning method has been proposed for load demand forecasting.The hybrid method composes of Empirical Mode Decomposition and Deep Belief Network.Empirical Mode Decomposition based methods outperform the single structure models.Deep learning shows more advantages when the forecasting horizon increases. Load demand forecasting is a critical process in the planning of electric utilities. An ensemble method composed of Empirical Mode Decomposition (EMD) algorithm and deep learning approach is presented in this work. For this purpose, the load demand series were first decomposed into several intrinsic mode functions (IMFs). Then a Deep Belief Network (DBN) including two restricted Boltzmann machines (RBMs) was used to model each of the extracted IMFs, so that the tendencies of these IMFs can be accurately predicted. Finally, the prediction results of all IMFs can be combined by either unbiased or weighted summation to obtain an aggregated output for load demand. The electricity load demand data sets from Australian Energy Market Operator (AEMO) are used to test the effectiveness of the proposed EMD-based DBN approach. Simulation results demonstrated attractiveness of the proposed method compared with nine forecasting methods.

343 citations


Journal ArticleDOI
Tongwen Li1, Huanfeng Shen1, Qiangqiang Yuan1, Xuechen Zhang1, Liangpei Zhang1 
TL;DR: In this article, a geo-intelligent approach, which incorporates geographical correlation into an intelligent deep learning architecture, is developed to estimate PM2.5 in a deep belief network (denoted as Geoi-DBN).
Abstract: Fusing satellite observations and station measurements to estimate ground-level PM2.5 is promising for monitoring PM2.5 pollution. A geo-intelligent approach, which incorporates geographical correlation into an intelligent deep learning architecture, is developed to estimate PM2.5. Specifically, it considers geographical distance and spatiotemporally correlated PM2.5 in a deep belief network (denoted as Geoi-DBN). Geoi-DBN can capture the essential features associated with PM2.5 from latent factors. It was trained and tested with data from China in 2015. The results show that Geoi-DBN performs significantly better than the traditional neural network. The out-of-sample cross-validation R2 increases from 0.42 to 0.88, and RMSE decreases from 29.96 to 13.03 μg/m3. On the basis of the derived PM2.5 distribution, it is predicted that over 80% of the Chinese population live in areas with an annual mean PM2.5 of greater than 35 μg/m3. This study provides a new perspective for air pollution monitoring in large geographic regions.

299 citations


Journal ArticleDOI
TL;DR: A new diversified DBN is developed through regularizing pretraining and fine-tuning procedures by a diversity promoting prior over latent factors that obtain much better results than original DBNs and comparable or even better performances compared with other recent hyperspectral image classification methods.
Abstract: In the literature of remote sensing, deep models with multiple layers have demonstrated their potentials in learning the abstract and invariant features for better representation and classification of hyperspectral images. The usual supervised deep models, such as convolutional neural networks, need a large number of labeled training samples to learn their model parameters. However, the real-world hyperspectral image classification task provides only a limited number of training samples. This paper adopts another popular deep model, i.e., deep belief networks (DBNs), to deal with this problem. The DBNs allow unsupervised pretraining over unlabeled samples at first and then a supervised fine-tuning over labeled samples. But the usual pretraining and fine-tuning method would make many hidden units in the learned DBNs tend to behave very similarly or perform as “dead” (never responding) or “potential over-tolerant” (always responding) latent factors. These results could negatively affect description ability and thus classification performance of DBNs. To further improve DBN’s performance, this paper develops a new diversified DBN through regularizing pretraining and fine-tuning procedures by a diversity promoting prior over latent factors. Moreover, the regularized pretraining and fine-tuning can be efficiently implemented through usual recursive greedy and back-propagation learning framework. The experiments over real-world hyperspectral images demonstrated that the diversity promoting prior in both pretraining and fine-tuning procedure lead to the learned DBNs with more diverse latent factors, which directly make the diversified DBNs obtain much better results than original DBNs and comparable or even better performances compared with other recent hyperspectral image classification methods.

291 citations


Journal ArticleDOI
TL;DR: The effectiveness of the proposed, DNN-MRT technique is expressed by comparing statistical performance measures in terms of root mean squared error (RMSE), mean absolute error (MAE), and standard deviation error (SDE) with other existing techniques.

Proceedings ArticleDOI
10 Mar 2017
TL;DR: This paper bulided a simple Convolutional neural network on image classification and analyzed different methods of learning rate set and different optimization algorithm of solving the optimal parameters of the influence on image Classification.
Abstract: In recent years, deep learning has been used in image classification, object tracking, pose estimation, text detection and recognition, visual saliency detection, action recognition and scene labeling. Auto Encoder, sparse coding, Restricted Boltzmann Machine, Deep Belief Networks and Convolutional neural networks is commonly used models in deep learning. Among different type of models, Convolutional neural networks has been demonstrated high performance on image classification. In this paper we bulided a simple Convolutional neural network on image classification. This simple Convolutional neural network completed the image classification. Our experiments are based on benchmarking datasets minist [1] and cifar-10. On the basis of the Convolutional neural network, we also analyzed different methods of learning rate set and different optimization algorithm of solving the optimal parameters of the influence on image classification.

Journal ArticleDOI
TL;DR: An extensible deep belief network (DBN) based fault diagnosis model is proposed and individual fault features in both spatial and temporal domains are extracted by DBN sub-networks, aided by the mutual information technology.

Journal ArticleDOI
TL;DR: In this paper, a modification of an autoencoder type deep neural network was applied to the task of dimension reduction of molecular dynamics data, which can reliably find low-dimensional embeddings for high-dimensional feature spaces which capture the slow dynamics of the underlying stochastic processes.
Abstract: Inspired by the success of deep learning techniques in the physical and chemical sciences, we apply a modification of an autoencoder type deep neural network to the task of dimension reduction of molecular dynamics data. We can show that our time-lagged autoencoder reliably finds low-dimensional embeddings for high-dimensional feature spaces which capture the slow dynamics of the underlying stochastic processes - beyond the capabilities of linear dimension reduction techniques.

Journal ArticleDOI
Tongwen Li1, Huanfeng Shen1, Qiangqiang Yuan1, Xuechen Zhang1, Liangpei Zhang1 
TL;DR: In this article, a geo-intelligent approach, which incorporates geographical correlation into an intelligent deep learning architecture, is developed to estimate PM2.5 in a deep belief network (denoted as Geoi-DBN).
Abstract: Fusing satellite observations and station measurements to estimate ground-level PM2.5 is promising for monitoring PM2.5 pollution. A geo-intelligent approach, which incorporates geographical correlation into an intelligent deep learning architecture, is developed to estimate PM2.5. Specifically, it considers geographical distance and spatiotemporally correlated PM2.5 in a deep belief network (denoted as Geoi-DBN). Geoi-DBN can capture the essential features associated with PM2.5 from latent factors. It was trained and tested with data from China in 2015. The results show that Geoi-DBN performs significantly better than the traditional neural network. The cross-validation R increases from 0.63 to 0.94, and RMSE decreases from 29.56 to 13.68${\mu}$g/m3. On the basis of the derived PM2.5 distribution, it is predicted that over 80% of the Chinese population live in areas with an annual mean PM2.5 of greater than 35${\mu}$g/m3. This study provides a new perspective for air pollution monitoring in large geographic regions.

Journal ArticleDOI
TL;DR: A novel method called adaptive deep belief network (DBN) with dual-tree complex wavelet packet (DTCWPT) is developed and applied to the fault diagnosis of rolling bearings, confirming that the proposed method is more effective than the existing methods.
Abstract: Automatic and accurate identification of rolling bearing fault categories, especially for the fault severities and compound faults, is a challenge in rotating machinery fault diagnosis. For this purpose, a novel method called adaptive deep belief network (DBN) with dual-tree complex wavelet packet (DTCWPT) is developed in this paper. DTCWPT is used to preprocess the vibration signals to refine the fault characteristics information, and an original feature set is designed from each frequency-band signal of DTCWPT. An adaptive DBN is constructed to improve the convergence rate and identification accuracy with multiple stacked adaptive restricted Boltzmann machines (RBMs). The proposed method is applied to the fault diagnosis of rolling bearings. The results confirm that the proposed method is more effective than the existing methods.

Proceedings Article
26 Apr 2017
TL;DR: In this article, a tensor factorization-based deep multi-task representation learning framework is proposed to learn cross-task sharing structure at every layer in a deep network, which is based on generalizing the matrix factorization techniques explicitly or implicitly used by many conventional MTL algorithms to tensor factors.
Abstract: Most contemporary multi-task learning methods assume linear models. This setting is considered shallow in the era of deep learning. In this paper, we present a new deep multi-task representation learning framework that learns cross-task sharing structure at every layer in a deep network. Our approach is based on generalising the matrix factorisation techniques explicitly or implicitly used by many conventional MTL algorithms to tensor factorisation, to realise automatic learning of end-to-end knowledge sharing in deep networks. This is in contrast to existing deep learning approaches that need a user-defined multi-task sharing strategy. Our approach applies to both homogeneous and heterogeneous MTL. Experiments demonstrate the efficacy of our deep multi-task representation learning in terms of both higher accuracy and fewer design choices.

Proceedings ArticleDOI
24 May 2017
TL;DR: This paper deals with the field of computer vision, mainly for the application of deep learning in object detection task, and a new dataset is built according to those commonly used datasets.
Abstract: This paper deals with the field of computer vision, mainly for the application of deep learning in object detection task. On the one hand, there is a simple summary of the datasets and deep learning algorithms commonly used in computer vision. On the other hand, a new dataset is built according to those commonly used datasets, and choose one of the network called faster r-cnn to work on this new dataset. Through the experiment to strengthen the understanding of these networks, and through the analysis of the results learn the importance of deep learning technology, and the importance of the dataset for deep learning.

Posted Content
TL;DR: This paper primarily focuses on the precedents of the models above, examining how the initial ideas are assembled to construct the early models and how these preliminary models are developed into their current forms.
Abstract: This paper is a review of the evolutionary history of deep learning models. It covers from the genesis of neural networks when associationism modeling of the brain is studied, to the models that dominate the last decade of research in deep learning like convolutional neural networks, deep belief networks, and recurrent neural networks. In addition to a review of these models, this paper primarily focuses on the precedents of the models above, examining how the initial ideas are assembled to construct the early models and how these preliminary models are developed into their current forms. Many of these evolutionary paths last more than half a century and have a diversity of directions. For example, CNN is built on prior knowledge of biological vision system; DBN is evolved from a trade-off of modeling power and computation complexity of graphical models and many nowadays models are neural counterparts of ancient linear models. This paper reviews these evolutionary paths and offers a concise thought flow of how these models are developed, and aims to provide a thorough background for deep learning. More importantly, along with the path, this paper summarizes the gist behind these milestones and proposes many directions to guide the future research of deep learning.

Journal ArticleDOI
TL;DR: The results proved that the accuracy achieved by Deep Boltzmann Machines, Deep Belief Networks and Stacked Auto-Encoders are highly reliable and applicable in fault diagnosis of rolling bearing.

Journal ArticleDOI
TL;DR: This Review Paper highlights latest studies regarding the implementation of deep learning models such as deep neural networks, convolutional neural networks and many more for solving different problems of sentiment analysis such as sentiment classification, cross lingual problems, textual and visual analysis and product review analysis, etc.
Abstract: The World Wide Web such as social networks, forums, review sites and blogs generate enormous heaps of data in the form of users views, emotions, opinions and arguments about different social events, products, brands, and politics. Sentiments of users that are expressed on the web has great influence on the readers, product vendors and politicians. The unstructured form of data from the social media is needed to be analyzed and well-structured and for this purpose, sentiment analysis has recognized significant attention. Sentiment analysis is referred as text organization that is used to classify the expressed mind-set or feelings in different manners such as negative, positive, favorable, unfavorable, thumbs up, thumbs down, etc. The challenge for sentiment analysis is lack of sufficient labeled data in the field of Natural Language Processing (NLP). And to solve this issue, the sentiment analysis and deep learning techniques have been merged because deep learning models are effective due to their automatic learning capability. This Review Paper highlights latest studies regarding the implementation of deep learning models such as deep neural networks, convolutional neural networks and many more for solving different problems of sentiment analysis such as sentiment classification, cross lingual problems, textual and visual analysis and product review analysis, etc.

Journal ArticleDOI
TL;DR: The proposed method significantly improves the accuracy of power transformer fault diagnosis by analyzing the relationship between the gases dissolved in transformer oil and fault types and the Non­code ratios of the gases are determined as the characterizing parameter of the DBN model.
Abstract: Dissolved gas analysis (DGA) of insulating oil can provide an important basis for transformer fault diagnosis. To improve diagnosis accuracy, this paper presents a new transformer fault diagnosis method based on deep belief networks (DBN). By analyzing the relationship between the gases dissolved in transformer oil and fault types, the Non­code ratios of the gases are determined as the characterizing parameter of the DBN model. DBN adopts multi-layer and multi-dimension mapping to extract more detailed differences of fault types. In this process, the diagnosis parameters are pre-trained. A back-propagation algorithm adjusts them with the labels of the samples and optimizes the parameters. To verify the effect of the proposed method, the diagnostic DBN model is constructed and tested using various oil chromatographic datasets collected from the State Grid Corporation of China and previous publications. The performances of the DBN diagnosis model are analyzed by different characterizing parameters, different training datasets and sample datasets. In addition, the influence of discharge and overheating multiple faults on the diagnosis model is studied. The performance of the proposed approach is compared with that derived from support vector machine (SVM), back-propagation neural network (BPNN) and ratio methods respectively. The results show that the proposed method significantly improves the accuracy of power transformer fault diagnosis.

Journal ArticleDOI
TL;DR: The experimental results on the Interactive Emotional Dyadic Motion Capture and Sustained Emotionally Colored Machine-Human Interaction Using Nonverbal Expression databases show significant improvements on unweighted accuracy, illustrating the benefit of utilizing additional information in a multi-task learning setup for emotion recognition.
Abstract: Dimensional models have been proposed in psychology studies to represent complex human emotional expressions. Activation and valence are two common dimensions in such models. They can be used to describe certain emotions. For example, anger is one type of emotion with a low valence and high activation value; neutral has both a medium level valence and activation value. In this work, we propose to apply multi-task learning to leverage activation and valence information for acoustic emotion recognition based on the deep belief network (DBN) framework. We treat the categorical emotion recognition task as the major task. For the secondary task, we leverage activation and valence labels in two different ways, category level based classification and continuous level based regression. The combination of the loss functions from the major and secondary tasks is used as the objective function in the multi-task learning framework. After iterative optimization, the values from the last hidden layer in the DBN are used as new features and fed into a support vector machine classifier for emotion recognition. Our experimental results on the Interactive Emotional Dyadic Motion Capture and Sustained Emotionally Colored Machine-Human Interaction Using Nonverbal Expression databases show significant improvements on unweighted accuracy, illustrating the benefit of utilizing additional information in a multi-task learning setup for emotion recognition.

Journal ArticleDOI
TL;DR: A deep learning-based framework by using a novel hybrid convolutional neural network and deep belief network to predict the RBP interaction sites and motifs on RNAs, which can achieve promising performance and easily capture interpretable binding motifs.
Abstract: RNAs play key roles in cells through the interactions with proteins known as the RNA-binding proteins (RBP) and their binding motifs enable crucial understanding of the post-transcriptional regulation of RNAs. How the RBPs correctly recognize the target RNAs and why they bind specific positions is still far from clear. Machine learning-based algorithms are widely acknowledged to be capable of speeding up this process. Although many automatic tools have been developed to predict the RNA-protein binding sites from the rapidly growing multi-resource data, e.g. sequence, structure, their domain specific features and formats have posed significant computational challenges. One of current difficulties is that the cross-source shared common knowledge is at a higher abstraction level beyond the observed data, resulting in a low efficiency of direct integration of observed data across domains. The other difficulty is how to interpret the prediction results. Existing approaches tend to terminate after outputting the potential discrete binding sites on the sequences, but how to assemble them into the meaningful binding motifs is a topic worth of further investigation. In viewing of these challenges, we propose a deep learning-based framework (iDeep) by using a novel hybrid convolutional neural network and deep belief network to predict the RBP interaction sites and motifs on RNAs. This new protocol is featured by transforming the original observed data into a high-level abstraction feature space using multiple layers of learning blocks, where the shared representations across different domains are integrated. To validate our iDeep method, we performed experiments on 31 large-scale CLIP-seq datasets, and our results show that by integrating multiple sources of data, the average AUC can be improved by 8% compared to the best single-source-based predictor; and through cross-domain knowledge integration at an abstraction level, it outperforms the state-of-the-art predictors by 6%. Besides the overall enhanced prediction performance, the convolutional neural network module embedded in iDeep is also able to automatically capture the interpretable binding motifs for RBPs. Large-scale experiments demonstrate that these mined binding motifs agree well with the experimentally verified results, suggesting iDeep is a promising approach in the real-world applications. The iDeep framework not only can achieve promising performance than the state-of-the-art predictors, but also easily capture interpretable binding motifs. iDeep is available at http://www.csbio.sjtu.edu.cn/bioinf/iDeep

Journal ArticleDOI
TL;DR: The study results showed the deep structured algorithms with automatically generated features can achieve desirable performance in lung nodule diagnosis with well-tuned parameters and large enough dataset, and the deep learning algorithms can have better performance than current popular CADx.

Proceedings ArticleDOI
01 Oct 2017
TL;DR: This study used Tensorflow, one of the most popular deep learning libraries to classify MNIST dataset, which is frequently used in data analysis studies and shows that the most accurate classification rate is obtained using the ReLu activation function.
Abstract: Deep learning is a subfield of machine learning which uses artificial neural networks that is inspired by the structure and function of the human brain. Despite being a very new approach, it has become very popular recently. Deep learning has achieved much higher success in many applications where machine learning has been successful at certain rates. In particular It is preferred in the classification of big data sets because it can provide fast and efficient results. In this study, we used Tensorflow, one of the most popular deep learning libraries to classify MNIST dataset, which is frequently used in data analysis studies. Using Tensorflow, which is an open source artificial intelligence library developed by Google, we have studied and compared the effects of multiple activation functions on classification results. The functions used are Rectified Linear Unit (ReLu), Hyperbolic Tangent (tanH), Exponential Linear Unit (eLu), sigmoid, softplus and softsign. In this Study, Convolutional Neural Network (CNN) and SoftMax classifier are used as deep learning artificial neural network. The results show that the most accurate classification rate is obtained using the ReLu activation function.

Posted ContentDOI
TL;DR: Several techniques for sampling and visualizing the latent spaces of generative models are introduced and two new techniques for deriving attribute vectors are demonstrated: bias-corrected vectors with data replication and synthetic vectors withData augmentation.
Abstract: We introduce several techniques for sampling and visualizing the latent spaces of generative models. Replacing linear interpolation with spherical linear interpolation prevents diverging from a model's prior distribution and produces sharper samples. J-Diagrams and MINE grids are introduced as visualizations of manifolds created by analogies and nearest neighbors. We demonstrate two new techniques for deriving attribute vectors: bias-corrected vectors with data replication and synthetic vectors with data augmentation. Binary classification using attribute vectors is presented as a technique supporting quantitative analysis of the latent space. Most techniques are intended to be independent of model type and examples are shown on both Variational Autoencoders and Generative Adversarial Networks.

Proceedings ArticleDOI
19 Aug 2017
TL;DR: This work addresses the problem via label distribution learning and develops a multi-task deep framework by jointly optimizing classification and distribution prediction by exploiting two weak forms of prior knowledge, which are expressed as similarity information between labels, to generate emotional distribution for each category.
Abstract: Visual sentiment analysis is attracting more and more attention with the increasing tendency to express emotions through visual contents. Recent algorithms in Convolutional Neural Networks (CNNs) considerably advance the emotion classification, which aims to distinguish differences among emotional categories and assigns a single dominant label to each image. However, the task is inherently ambiguous since an image usually evokes multiple emotions and its annotation varies from person to person. In this work, we address the problem via label distribution learning and develop a multi-task deep framework by jointly optimizing classification and distribution prediction. While the proposed method prefers to the distribution datasets with annotations of different voters, the majority voting scheme is widely adopted as the ground truth in this area, and few dataset has provided multiple affective labels. Hence, we further exploit two weak forms of prior knowledge, which are expressed as similarity information between labels, to generate emotional distribution for each category. The experiments conducted on both distribution datasets, i.e. Emotion6, Flickr LDL, Twitter LDL, and the largest single label dataset, i.e. Flickr and Instagram, demonstrate the proposed method outperforms the state-of-the-art approaches.

Journal ArticleDOI
TL;DR: Some of the libraries and tools that are available to aid in the construction and efficient execution of deep learning as applied to medical images are described.
Abstract: Deep learning is an important new area of machine learning which encompasses a wide range of neural network architectures designed to complete various tasks. In the medical imaging domain, example tasks include organ segmentation, lesion detection, and tumor classification. The most popular network architecture for deep learning for images is the convolutional neural network (CNN). Whereas traditional machine learning requires determination and calculation of features from which the algorithm learns, deep learning approaches learn the important features as well as the proper weighting of those features to make predictions for new data. In this paper, we will describe some of the libraries and tools that are available to aid in the construction and efficient execution of deep learning as applied to medical images.

Journal ArticleDOI
TL;DR: This paper introduces a trained model based on a DBN to classify 4100 peripheral blood smear images into the parasite or non-parasite class using a deep belief network (DBN).
Abstract: In this paper, we propose a novel method to identify the presence of malaria parasites in human peripheral blood smear images using a deep belief network (DBN). This paper introduces a trained model based on a DBN to classify 4100 peripheral blood smear images into the parasite or non-parasite class. The proposed DBN is pre-trained by stacking restricted Boltzmann machines using the contrastive divergence method for pre-training. To train the DBN, we extract features from the images and initialize the visible variables of the DBN. A concatenated feature of color and texture is used as a feature vector in this paper. Finally, the DBN is discriminatively fine-tuned using a backpropagation algorithm that computes the probability of class labels. The optimum size of the DBN architecture used in this paper is 484-600-600-600-600-2, in which the visible layer has 484 nodes and the output layer has two nodes with four hidden layers containing 600 hidden nodes in every layer. The proposed method has performed significantly better than the other state-of-the-art methods with an F-score of 89.66%, a sensitivity of 97.60%, and specificity of 95.92%. This paper is the first application of a DBN for malaria parasite detection in human peripheral blood smear images.