We propose an independence criterion based on the eigen-spectrum of covariance operators in reproducing kernel Hilbert spaces (RKHSs), consisting of an empirical estimate of the Hilbert-Schmidt norm of the cross-covariance operator (we term this a Hilbert-Schmidt Independence Criterion, or HSIC). This approach has several advantages, compared with previous kernel-based independence criteria. First, the empirical estimate is simpler than any other kernel dependence test, and requires no user-defined regularisation. Second, there is a clearly defined population quantity which the empirical estimate approaches in the large sample limit, with exponential convergence guaranteed between the two: this ensures that independence tests based on HSIC do not suffer from slow learning rates. Finally, we show in the context of independent component analysis (ICA) that the performance of HSIC is competitive with that of previously published kernel-based criteria, and of other recently published ICA methods.

Measuring statistical dependence with Hilbert-Schmidt norms

We present a new algorithm for manifold learning and nonlinear dimensionality reduction. Based on a set of unorganized da-ta points sampled with noise from a parameterized manifold, the local geometry of the manifold is learned by constructing an approxi-mation for the tangent space at each point, and those tangent spaces are then aligned to give the global coordinates of the data pointswith respect to the underlying manifold. We also present an error analysis of our algorithm showing that reconstruction errors can bequite small in some cases. We illustrate our algorithm using curves and surfaces both in 2D/3D Euclidean spaces and higher dimension-al Euclidean spaces. We also address several theoretical and algorithmic issues for further research and improvements.

Principal Manifolds and Nonlinear Dimensionality Reduction via Tangent Space Alignment

The world continues to generate quintillion bytes of data daily, leading to the pressing needs for new efforts in dealing with the grand challenges brought by Big Data. Today, there is a growing consensus among the computational intelligence communities that data volume presents an immediate challenge pertaining to the scalability issue. However, when addressing volume in Big Data analytics, researchers in the data analytics community have largely taken a one-sided study of volume, which is the "Big Instance Size" factor of the data. The flip side of volume which is the dimensionality factor of Big Data, on the other hand, has received much lesser attention. This article thus represents an attempt to fill in this gap and places special focus on this relatively under-explored topic of "Big Dimensionality", wherein the explosion of features (variables) brings about new challenges to computational intelligence. We begin with an analysis on the origins of Big Dimensionality. The evolution of feature dimensionality in the last two decades is then studied using popular data repositories considered in the data analytics and computational intelligence research communities. Subsequently, the state-of-the-art feature selection schemes reported in the field of computational intelligence are reviewed to reveal the inadequacies of existing approaches in keeping pace with the emerging phenomenon of Big Dimensionality. Last but not least, the "curse and blessing of Big Dimensionality" are delineated and deliberated.

https://dr.ntu.edu.sg/bitstream/10220/39675/1/The%20Emerging%20%e2%80%9cBig%20Dimensionality.pdf

The Emerging Big Dimensionality

The recent literature indicates that preserving global pairwise sample similarity is of great importance for feature selection and that many existing selection criteria essentially work in this way. In this paper, we argue that besides global pairwise sample similarity, the local geometric structure of data is also critical and that these two factors play different roles in different learning scenarios. In order to show this, we propose a global and local structure preservation framework for feature selection (GLSPFS) which integrates both global pairwise sample similarity and local geometric data structure to conduct feature selection. To demonstrate the generality of our framework, we employ methods that are well known in the literature to model the local geometric data structure and develop three specific GLSPFS-based feature selection algorithms. Also, we develop an efficient optimization algorithm with proven global convergence to solve the resulting feature selection problem. A comprehensive experimental study is then conducted in order to compare our feature selection algorithms with many state-of-the-art ones in supervised, unsupervised, and semisupervised learning scenarios. The result indicates that: 1) our framework consistently achieves statistically significant improvement in selection performance when compared with the currently used algorithms; 2) in supervised and semisupervised learning scenarios, preserving global pairwise similarity is more important than preserving local geometric data structure; 3) in the unsupervised scenario, preserving local geometric data structure becomes clearly more important; and 4) the best feature selection performance is always obtained when the two factors are appropriately integrated. In summary, this paper not only validates the advantages of the proposed GLSPFS framework but also gains more insight into the information to be preserved in different feature selection tasks.

Global and Local Structure Preservation for Feature Selection

In this thesis, we study the problem of face recognition based on image sets. The main objective of our work is to develop set-based distance metrics that are able to measure the similarity between image sets, rather than conventional distance metrics that can only measure the distance between samples. The face images obtained from real-life impose great challenges to the conventional face recognition systems. Large variations in appearances and various imperfections such as occlusions and misalignments in the face images severely degrade the recognition performance. One possible solution is to utilize more face images for each person, e.g., a collection of photos from personal galleries or frames extracted from a video clip. Under such circumstances, the face recognition task becomes the process of modelling and matching image sets. Our investigation then focuses on developing appropriate models and set-based distance metrics for representing different image sets.

Face recognition based on image sets

Histogram of oriented gradients (HOG) descriptor was initially applied to human detection and achieved great success. In recent years, HOG descriptor has also been applied to face recognition. However, comparing with other sophisticated feature descriptors such as LBP, Gabor and so on, there are still considerable research space on the application of HOG features for face recognition. There are two main contributions. On one hand, the main parameters are statistically analysed characterising HOG descriptor for face recognition, which seems to be not discussed clearly in literatures so far. On the other hand, a novel framework for face recognition based on the fusion of global and local HOG features has been proposed. Face images are first illumination normalised by the DoG filter. Secondly, global and local HOG features are extracted by PCA + LDA or LDA with different framework. Finally, in decision level, global and local classifiers are built by the nearest neighbour classifier, after that, two classifiers are fused by a weighted sum rule. Experimental results on two large-scale face databases FERET and CAS-PEAL-R1 show that, in comparison with 12 state-of-the-art approaches of face recognition, the proposed method achieves the highest average recognition rate.

https://ietresearch.onlinelibrary.wiley.com/doi/pdf/10.1049/iet-cvi.2012.0302

Face recognition based on the fusion of global and local HOG features of face images

Dimensionality reduction is vital in many fields, and alignment-based methods for nonlinear dimensionality reduction have become popular recently because they can map the high-dimensional data into a low-dimensional subspace with the property of local isometry. However, the relationships between patches in original high-dimensional space cannot be ensured to be fully preserved during the alignment process. In this paper, we propose a novel method for nonlinear dimensionality reduction called local coordinates alignment with global preservation. We first introduce a reasonable definition of topology-preserving landmarks (TPLs), which not only contribute to preserving the global structure of datasets and constructing a collection of overlapping linear patches, but they also ensure that the right landmark is allocated to the new test point. Then, an existing method for dimensionality reduction that has good performance in preserving the global structure is used to derive the low-dimensional coordinates of TPLs. Local coordinates of each patch are derived using tangent space of the manifold at the corresponding landmark, and then these local coordinates are aligned into a global coordinate space with the set of landmarks in low-dimensional space as reference points. The proposed alignment method, called landmarks-based alignment, can produce a closed-form solution without any constraints, while most previous alignment-based methods impose the unit covariance constraint, which will result in the deficiency of global metrics and undesired rescaling of the manifold. Experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithm.

Local Coordinates Alignment With Global Preservation for Dimensionality Reduction

Grassmann manifold for nearest points image set classification

The goal of nonlinear dimensionality reduction is to find the meaningful low dimensional structure of the nonlinear manifold from the high dimensional data. As a classic method of nonlinear dimensional reduction, locally linear embedding (LLE) is more and more attractive to researchers due to its ability to deal with large amounts of high dimensional data and its noniterative way of finding the embeddings. However, several problems in the LLE algorithm still remain open, such as its sensitivity to noise, inevitable ill-conditioned eigenproblems, the inability to deal with the novel data, etc. The existing extensions are comprehensively reviewed and discussed classifying into different categories in this paper. Their strategies, advantages/disadvantages and performances are elaborated. By generalizing different tactics in various extensions related to different stages of LLE and evaluating their performances, several promising directions for future research have been suggested.

Locally linear embedding: a review

Among the representative algorithms of manifold learning, Hessian locally linear embedding (HLLE) and local tangent space alignment (LTSA) algorithms haven been regarded as two different algorithms. However, in practice, the effects of these two algorithms are very similar and LTSA performs better than HLLE in some applications. This paper tries to account for this phenomenon from a mathematical point of view. There are only two differences between HLLE and LTSA. First, LTSA includes a data point into its neighborhood, while HLLE does not. Second, HLLE and LTSA use different methods to align the local coordinates of manifold. In this paper, we show that, the first difference between HLLE and LTSA is not essential. However, from the viewpoint of data utilization, LTSA does better than HLLE in the neighborhood construction. This may account for why LTSA can perform better than HLLE in some applications. As for the second difference between HLLE and LTSA, we first prove that, the alignment equations used by HLLE and LTSA are exactly the same. Second, we prove that, although HLLE and LTSA uses different methods to solve the alignment equation, their solutions are exactly the same, provided that HLLE adopts the same method as LTSA to construct the neighborhoods. Based on these arguments, we claim that HLLE and LTSA are equivalent to each other. This conclusion can also be verified experimentally by using manifold learning MATLAB demo (MANI), a widely-used experimental platform of manifold learning. When testing HLLE on MANI, if HLLE adopts the same method as LTSA to construct the neighborhoods, the experimental results presented by MANI will be the same as those of LTSA.

Zhengming Ma

Papers

Face recognition based on the fusion of global and local HOG features of face images

Local Coordinates Alignment With Global Preservation for Dimensionality Reduction

Grassmann manifold for nearest points image set classification

Locally linear embedding: a review

On the Equivalence of HLLE and LTSA