Showing papers on "Autoencoder published in 2009"

Measuring Invariances in Deep Networks

Abstract: For many pattern recognition tasks, the ideal input feature would be invariant to multiple confounding properties (such as illumination and viewing angle, in computer vision applications). Recently, deep architectures trained in an unsupervised manner have been proposed as an automatic method for extracting useful features. However, it is difficult to evaluate the learned features by any means other than using them in a classifier. In this paper, we propose a number of empirical tests that directly measure the degree to which these learned features are invariant to different input transformations. We find that stacked autoencoders learn modestly increasingly invariant features with depth when trained on natural images. We find that convolutional deep belief networks learn substantially more invariant features in each layer. These results further justify the use of "deep" vs. "shallower" representations, but suggest that mechanisms beyond merely stacking one autoencoder on top of another may be important for achieving invariance. Our evaluation metrics can also be used to evaluate future work in deep learning, and thus help the development of future algorithms.

A Deep Non-linear Feature Mapping for Large-Margin kNN Classification

Abstract: KNN is one of the most popular data mining methods for classification, but it often fails to work well with inappropriate choice of distance metric or due to the presence of numerous class-irrelevant features Linear feature transformation methods have been widely applied to extract class-relevant information to improve kNN classification, which is very limited in many applications Kernels have also been used to learn powerful non-linear feature transformations, but these methods fail to scale to large datasets In this paper, we present a scalable non-linear feature mapping method based on a deep neural network pretrained with Restricted Boltzmann Machines for improving kNN classification in a large-margin framework, which we call DNet-kNN DNet-kNN can be used for both classification and for supervised dimensionality reduction The experimental results on two benchmark handwritten digit datasets and one newsgroup text dataset show that DNet-kNN has much better performance than large-margin kNN using a linear mapping and kNN based on a deep autoencoder pretrained with Restricted Boltzmann Machines

System for determining most probable cause of a problem in a plant

Abstract: A system for determining a most probable cause or causes of a problem in a plant is disclosed. The system includes a plant, the plant having a plurality of subsystems that contribute to the operation of the plant, the plurality of subsystems having operating functions that produce operational signals. A plurality of sensors that are operable to detect the operational signals from the plurality of subsystems and transmit data related to the signals is also provided. An advisory system is disclosed that receives an input, the input being in the form of data from the plurality of sensors, possible input root causes of the problem, possible input symptoms of the problem and/or combinations thereof. The advisory system has an autoencoder in the form of a recurrent neural network. The recurrent neural network has sparse connectivity in a plurality of nodes, and the autoencoder is also operable to receive the input and perform multiple iterations of computations at each of the plurality of nodes as a function of the input and provide an output. The output can be in the form of possible output causes of the problem, possible output symptoms of the problem and/or combinations thereof.

Missing Data Estimation using Principle Component Analysis and Autoassociative Neural Networks

Abstract: Three new methods are used for estimating missing data in a database using Neural Networks, Principal Component Analysis and Genetic Algorithms are presented. The proposed methods are tested on a set of data obtained from the South African Antenatal Survey. The data is a collection of demographic properties of patients. The proposed methods use Principal Component Analysis to remove redundancies and reduce the dimensionality in the data. Variations of autoassociative Neural Networks are used to further reduce the dimensionality of the data. A Genetic Algorithm is then used to find the missing data by optimizing the error function of the three variants of the Autoencoder Neural Network. The proposed system was tested on data with 1 to 6 missing fields in a single record of data and the accuracy of the estimated values were calculated and recorded. All methods are as accurate as a conventional feedforward neural network structure however the use of the newly proposed methods employs neural network architectures that have fewer hidden nodes.

AUTOENCODER NEURAL NETWORKS: A Performance Study Based on Image Reconstruction, Recognition and Compression

Abstract: Autoencoders are feedforward neural networks which can have more than one hidden layer. These networks attempt to reconstruct the input data at the output layer. Since the size of the hidden layer in the autoencoders is smaller than the size of the input data, the dimensionality of input data is reduced to a smaller-dimensional code space at the hidden layer. However, training a multilayer autoencoder is tedious. This is due to the fact that the weights at deep hidden layers are hardly optimized. The research work has focused on the characteristics, training and performance evaluation of autoencoders. The concepts of stacking and Restricted Boltzmann Machine have also been discussed in detail. Two datasets, namely ORL face dataset and MNIST handwritten digit dataset have been employed in these experiments. The performances of the autoencoders have also been compared with that of PCA. It has been shown that the autoencoders can also be used for image compression. The compression efficiency has been studied using DDSM dataset (mammogram dataset). Since image patches were used for training, it was possible to compress and decompress mammograms of different sizes.

Large-Margin kNN Classification Using a Deep Encoder Network

Abstract: KNN is one of the most popular classification methods, but it often fails to work well with inappropriate choice of distance metric or due to the presence of numerous class-irrelevant features. Linear feature transformation methods have been widely applied to extract class-relevant information to improve kNN classification, which is very limited in many applications. Kernels have been used to learn powerful non-linear feature transformations, but these methods fail to scale to large datasets. In this paper, we present a scalable non-linear feature mapping method based on a deep neural network pretrained with restricted boltzmann machines for improving kNN classification in a large-margin framework, which we call DNet-kNN. DNet-kNN can be used for both classification and for supervised dimensionality reduction. The experimental results on two benchmark handwritten digit datasets show that DNet-kNN has much better performance than large-margin kNN using a linear mapping and kNN based on a deep autoencoder pretrained with retricted boltzmann machines.

Etude de techniques d'apprentissage non-supervise pour l'amelioration de l'entrainement supervise de modeles connexionnistes

Abstract: The objective of the field of artificial intelligence is the development of computer systems capable of simulating a behavior reminiscent of human intelligence. In particular, we would like to build a machine that would be able to solve tasks related to vision (e.g., object recognition), natural language (e.g., topic classification) or signal processing (e.g., speech recognition). The general approach developed in the sub-field of machine learning to solve such tasks is based on using labeled data to train a model to emulate the desired behavior. One such model that has been proposed is the artificial neural network, which can adapt its behavior using a backpropagated gradient [103, 135] that is informative of the errors made by the network. Popular during the 80's, this specific approach has since lost some of its appeal, following the development of kernel methods. Indeed, kernel methods are often found to be more stable, easier to use, and their performance usually compares favorably on a vast range of problems. Since the foundation of the field, machine learning methods have progressed not only in their inner workings, but also in the complexity of problems they can tackle. More recently however, it has been argued [12, 15] that kernel methods might not be able to solve efficiently enough problems of the complexity that is expected from artificial intelligence. At the same time, Hinton et al. [84] put forth a breakthrough in neural network training, by developing a procedure able to train more complex neural networks (i.e., with more layers of hidden neurons) than previously possible. This is the context in which the work presented in this thesis started. This thesis begins with the introduction of the basic principles of machine learning (Chapter 1) as well as the known obstacles to achieve good generalization performance (Chapter 2). Then, the work from five papers is presented, with each of these papers' contribution relying on a form of unsupervised learning. The first paper (Chapter 4) presents a training method for a specific form of single hidden layer neural network (the Restricted Boltzmann Machine), based on the combination of supervised and unsupervised learning. This method achieves a better generalization performance than a standard neural network and a kernel support vector machine. This observation emphasizes the beneficial effect of unsupervised learning for training neural networks. Then, the second paper (Chapter 6) studies and extends the training procedure of Hinton et al. [84]. More specifically, we propose a different but more flexible approach for initializing a deep (i.e., with many hidden layers) neural network, based on autoassociator networks. We also empirically analyze the impact of varying the number of layers and number of hidden neurons on the performance of a neural network, and we describe variants of the same training procedure that are more appropriate for continuous-valued inputs and online learning. The third paper (Chapter 8) describes a more intensive empirical evaluation of training algorithms for deep networks, on several classification problems. These problems have been generated based on several factors of variations, in order to simulate a property that is expected from artificial intelligence problems. The experiments presented in this paper tend to show that deep networks are more appropriate than shallow models, such as kernel methods. The fourth paper (Chapter 10) develops an improved variation of the autoassociator network. This simple variation, which brings better generalization performance to deep networks, modifies the autoassociator network's training procedure by corrupting its input and forcing the network to denoise it. The fifth and final paper (Chapter 12) contributes another improved variation on autoassociator networks, by allowing inhibitory/facilitatory interactions between the hidden layer neurons. We show that such interactions can be learned and can be beneficial to the performance of deep networks. Keywords: unsupervised learning, neural network, Restricted Boltzmann Machine, autoassociator, autoencoder, deep architecture, deep learning

A Novel Tracking and Recognition Algorithm Using "Continuous Autoencoder" Network

Abstract: Motion trajectory is one of the most important cues for tracking and behavior recognition and can be widely applied to numerous fields such as visual surveillance and guidance. However, it is a difficult problem to directly model the spatiotemporal variations of trajectories due to their high dimensionality and nonlinearity. In this paper, a novel tracking and trajectory recognition algorithm is proposed by combining a bi-directional deep neural network called “Continuous Autoencoder” into a probabilistic framework. First, the “Continuous Autoencoder” network embeds high-dimensional trajectories in a two-dimensional plane based on a peculiar training rule and learns a trajectory generative model by its inverse mapping. Then a set of plausible trajectories are generated by the trajectory generative model. In the tracking process, the target state at each time step is estimated by combining above plausible trajectory set with particle filter. The trajectory identity is inferred by evaluating the improved Hausdorff distance between the estimated trajectory up to now and the truncated reference trajectories. Furthermore, the trajectory recognition results can provide valuable information for the next tracking. The experiments on tracking and recognizing handwritten digits show that the proposed algorithm can not only robustly track and exactly recognize in background clutter and occlusion, but also realize the track before identification. Keywords-“Continuous Autoencoder” network; nonlinear dimensionality reduction; particle filter; trajectory tracking and recognition

HIV analysis using computational intelligence

Abstract: In this study, a new method to analyze HIV using a combination of autoencoder networks and genetic algorithms is proposed. The proposed method is tested on a set of demographic properties of individuals obtained from the South African antenatal survey. The autoencoder model is then compared with a conventional feedforward neural network model and yields a classification accuracy of 92% compared to 84% obtained for the conventional feedforward model. The autoencoder model is then used to propose a new method of approximating missing entries in the HIV database using ant colony optimization. This method is able to estimate missing input to an accuracy of 80%. The estimated missing input values are then used to analyze HIV. The autoencoder network classifier model yields a classification accuracy of 81% in the presence of missing input values. The feedforward neural network classifier model yields a classification accuracy of 82% in the presence of missing input values. A control mechanism is proposed to assess the effect of demographic properties on the HIV status of individuals, based on inverse neural networks, and autoencoder networks-based-on-genetic algorithms. This control mechanism is aimed at understanding whether HIV susceptibility can be controlled ii by modifying some of the demographic properties. The inverse neural network control model has accuracies of 77% and 82%, meanwhile the genetic algorithm model has accuracies of 77% and 92%, for the prediction of educational level of individuals, and gravidity, respectively. HIV modelling using neuro-fuzzy models is then investigated, and rules are extracted, which provide more valuable insight. The classification accuracy obtained by the neuro-fuzzy model is 86%. A rough set approximation is then investigated for rule extraction, and it is found that the rules present simplistic and understandable relationships on how the demographic properties affect HIV risk. The study concludes by investigating a model for automatic relevance determination, to determine which of the demographic properties is important for HIV modelling. A comparison is done between using the full input data set and the data set using the input parameters selected by the technique for the HIV classification. Age of the individual, gravidity, province, region, reported pregnancy and educational level were amongst the input parameters selected as relevant for classification of an individual’s HIV risk. This study thus proposes models, which can be used to understand HIV dynamics, and can be used by policy-makers to more effectively understand the demographic influences driving HIV infection.