We present a new algorithm for manifold learning and nonlinear dimensionality reduction. Based on a set of unorganized da-ta points sampled with noise from a parameterized manifold, the local geometry of the manifold is learned by constructing an approxi-mation for the tangent space at each point, and those tangent spaces are then aligned to give the global coordinates of the data pointswith respect to the underlying manifold. We also present an error analysis of our algorithm showing that reconstruction errors can bequite small in some cases. We illustrate our algorithm using curves and surfaces both in 2D/3D Euclidean spaces and higher dimension-al Euclidean spaces. We also address several theoretical and algorithmic issues for further research and improvements.

Principal Manifolds and Nonlinear Dimensionality Reduction via Tangent Space Alignment

This survey investigates current techniques for representing qualitative data for use as input to neural networks. Techniques for using qualitative data in neural networks are well known. However, researchers continue to discover new variations or entirely new methods for working with categorical data in neural networks. Our primary contribution is to cover these representation techniques in a single work. Practitioners working with big data often have a need to encode categorical values in their datasets in order to leverage machine learning algorithms. Moreover, the size of data sets we consider as big data may cause one to reject some encoding techniques as impractical, due to their running time complexity. Neural networks take vectors of real numbers as inputs. One must use a technique to map qualitative values to numerical values before using them as input to a neural network. These techniques are known as embeddings, encodings, representations, or distributed representations. Another contribution this work makes is to provide references for the source code of various techniques, where we are able to verify the authenticity of the source code. We cover recent research in several domains where researchers use categorical data in neural networks. Some of these domains are natural language processing, fraud detection, and clinical document automation. This study provides a starting point for research in determining which techniques for preparing qualitative data for use with neural networks are best. It is our intention that the reader should use these implementations as a starting point to design experiments to evaluate various techniques for working with qualitative data in neural networks. The third contribution we make in this work is a new perspective on techniques for using categorical data in neural networks. We organize techniques for using categorical data in neural networks into three categories. We find three distinct patterns in techniques that identify a technique as determined, algorithmic, or automated. The fourth contribution we make is to identify several opportunities for future research. The form of the data that one uses as an input to a neural network is crucial for using neural networks effectively. This work is a tool for researchers to find the most effective technique for working with categorical data in neural networks, in big data settings. To the best of our knowledge this is the first in-depth look at techniques for working with categorical data in neural networks.

/pdf/survey-on-categorical-data-for-neural-networks-28h02g6ago.pdf

Survey on categorical data for neural networks

IEEE transactions on systems, man and cybernetics. Part B, Cybernetics

Deep learning-based feature engineering methods for improved building energy prediction

Recurrent neural networks for real-time prediction of TBM operating parameters

An overview on data representation learning: From traditional feature learning to recent deep learning

Since Pearson developed principal component analysis (PCA) in 1901, feature learning (or called representation learning) has been studied for more than 100 years. During this period, many “shallow” feature learning methods have been proposed based on various learning criteria and techniques, until the popular deep learning research in recent years. In this advanced review, we describe the historical profile of the shallow feature learning research and introduce the important developments of the deep learning models. Particularly, we survey the deep architectures with benefits from the optimization of their width and depth, as these models have achieved new records in many applications, such as image classification and object detection. Finally, several interesting directions of deep learning are presented and briefly discussed.

From shallow feature learning to deep learning: Benefits from the width and depth of deep architectures

Softmax regressor is arguably the most commonly used classifier in convolutional neural networks (CNNs). However, the cross-entropy based softmax loss only supervises the deep neural networks to learn effective representations of data, but does not explicitly enforce the separability between the classes. In this paper, we propose a novel convolutional neural network model, called convolutional discriminative analysis (CDA). Beyond the softmax loss, CDA employs a convolutional discriminant loss (CD-Loss), which minimizes the distance between the sample and its class center while maximizes the distance between the sample and its adversarial class center in the space of the learned deep representations. Extensive experiments on two benchmark data sets, Fashion-MNIST and CIFAR-10, demonstrate the superiority of CDA over traditional deep CNNs on the image classification tasks.

Convolutional Discriminant Analysis

Many dimensionality reduction problems can be formulated as a trace ratio form, i.e. \(\hbox {argmax}_\mathbf{W}Tr(\mathbf{W}^T \mathbf{S}_p \mathbf{W}) / Tr(\mathbf{W}^T \mathbf{S}_t \mathbf{W})\), where \(\mathbf{S}_p\) and \(\mathbf{S}_t\) represent the (dis)similarity between data, \(\mathbf{W}\) is the projection matrix, and \(Tr(\cdot )\) is the trace of a matrix. Some representative algorithms of this category include principal component analysis (PCA), linear discriminant analysis (LDA) and marginal Fisher analysis (MFA). Previous research focuses on how to solve the trace ratio problems with either (generalized) eigenvalue decomposition or iterative algorithms. In this paper, we analyze an algorithm that transforms the trace ratio problems into a series of trace difference problems, i.e. \(\hbox {argmax}_\mathbf{W}Tr[(\mathbf{W}^T (\mathbf{S}_p - \lambda \mathbf{S}_t )\mathbf{W}]\), and propose the necessary and sufficient conditions for the existence of the optimal solution of trace ratio problems. The correctness of this theoretical result is proved. To evaluate the applied algorithm, we tested it on three face recognition applications. Experimental results demonstrate its convergence and effectiveness.

Xiao Ling

Papers

An overview on data representation learning: From traditional feature learning to recent deep learning

From shallow feature learning to deep learning: Benefits from the width and depth of deep architectures

Convolutional Discriminant Analysis

The Necessary and Sufficient Conditions for the Existence of the Optimal Solution of Trace Ratio Problems

Recurrent attention unit: A new gated recurrent unit for long-term memory of important parts in sequential data