Scientists working with large volumes of high-dimensional data, such as global climate patterns, stellar spectra, or human gene distributions, regularly confront the problem of dimensionality reduction: finding meaningful low-dimensional structures hidden in their high-dimensional observations. The human brain confronts the same problem in everyday perception, extracting from its high-dimensional sensory inputs-30,000 auditory nerve fibers or 10(6) optic nerve fibers-a manageably small number of perceptually relevant features. Here we describe an approach to solving dimensionality reduction problems that uses easily measured local metric information to learn the underlying global geometry of a data set. Unlike classical techniques such as principal component analysis (PCA) and multidimensional scaling (MDS), our approach is capable of discovering the nonlinear degrees of freedom that underlie complex natural observations, such as human handwriting or images of a face under different viewing conditions. In contrast to previous algorithms for nonlinear dimensionality reduction, ours efficiently computes a globally optimal solution, and, for an important class of data manifolds, is guaranteed to converge asymptotically to the true structure.

/pdf/a-global-geometric-framework-for-nonlinear-dimensionality-5brl02iqtj.pdf

A global geometric framework for nonlinear dimensionality reduction.

We develop a face recognition algorithm which is insensitive to large variation in lighting direction and facial expression. Taking a pattern classification approach, we consider each pixel in an image as a coordinate in a high-dimensional space. We take advantage of the observation that the images of a particular face, under varying illumination but fixed pose, lie in a 3D linear subspace of the high dimensional image space-if the face is a Lambertian surface without shadowing. However, since faces are not truly Lambertian surfaces and do indeed produce self-shadowing, images will deviate from this linear subspace. Rather than explicitly modeling this deviation, we linearly project the image into a subspace in a manner which discounts those regions of the face with large deviation. Our projection method is based on Fisher's linear discriminant and produces well separated classes in a low-dimensional subspace, even under severe variation in lighting and facial expressions. The eigenface technique, another method based on linearly projecting the image space to a low dimensional subspace, has similar computational requirements. Yet, extensive experimental results demonstrate that the proposed "Fisherface" method has error rates that are lower than those of the eigenface technique for tests on the Harvard and Yale face databases.

/pdf/eigenfaces-vs-fisherfaces-recognition-using-class-specific-2qyevb22n0.pdf

Eigenfaces vs. Fisherfaces: recognition using class specific linear projection

Is perception of the whole based on perception of its parts? There is psychological and physiological evidence for parts-based representations in the brain, and certain computational theories of object recognition rely on such representations. But little is known about how brains or computers might learn the parts of objects. Here we demonstrate an algorithm for non-negative matrix factorization that is able to learn parts of faces and semantic features of text. This is in contrast to other methods, such as principal components analysis and vector quantization, that learn holistic, not parts-based, representations. Non-negative matrix factorization is distinguished from the other methods by its use of non-negativity constraints. These constraints lead to a parts-based representation because they allow only additive, not subtractive, combinations. When non-negative matrix factorization is implemented as a neural network, parts-based representations emerge by virtue of two properties: the firing rates of neurons are never negative and synaptic strengths do not change sign.

Learning the parts of objects by non-negative matrix factorization

We consider the problem of automatically recognizing human faces from frontal views with varying expression and illumination, as well as occlusion and disguise. We cast the recognition problem as one of classifying among multiple linear regression models and argue that new theory from sparse signal representation offers the key to addressing this problem. Based on a sparse representation computed by C1-minimization, we propose a general classification algorithm for (image-based) object recognition. This new framework provides new insights into two crucial issues in face recognition: feature extraction and robustness to occlusion. For feature extraction, we show that if sparsity in the recognition problem is properly harnessed, the choice of features is no longer critical. What is critical, however, is whether the number of features is sufficiently large and whether the sparse representation is correctly computed. Unconventional features such as downsampled images and random projections perform just as well as conventional features such as eigenfaces and Laplacianfaces, as long as the dimension of the feature space surpasses certain threshold, predicted by the theory of sparse representation. This framework can handle errors due to occlusion and corruption uniformly by exploiting the fact that these errors are often sparse with respect to the standard (pixel) basis. The theory of sparse representation helps predict how much occlusion the recognition algorithm can handle and how to choose the training images to maximize robustness to occlusion. We conduct extensive experiments on publicly available databases to verify the efficacy of the proposed algorithm and corroborate the above claims.

/pdf/robust-face-recognition-via-sparse-representation-4bfqv6y74o.pdf

Robust Face Recognition via Sparse Representation

Learning parts of objects by non-negative matrix factorization

We present a system for automatic hot spots detection and segmentation in whole body FDG-PET images. The main contribution of our system is threefold. First, it has a novel body-section labeling module based on spatial hidden-Markov models (HMM); this allows different processing policies to be applied in different body sections. Second, the competition diffusion (CD) segmentation algorithm, which takes into account body-section information, converts the binary thresholding results to probabilistic interpretation and detects hot-spot region candidates. Third, a recursive intensity mode-seeking algorithm finds hot spot centers efficiently, and given these centers, a clinically meaningful protocol is proposed to accurately quantify hot spot volumes. Experimental results show that our system works robustly despite the large variations in clinical PET images.

/pdf/automatic-hot-spot-detection-and-segmentation-in-whole-body-2rxcii61ab.pdf

Automatic Hot Spot Detection and Segmentation in Whole Body FDG-PET Images

We propose an isometric self-organizing map (ISO-SOM) method for nonlinear dimensionality reduction, which integrates a self-organizing map model and an ISOMAP dimension reduction algorithm, organizing the high dimension data in a low dimension lattice structure. We apply the proposed method to the problem of appearance-based 3D hand posture estimation. As a learning stage, we use a realistic 3D hand model to generate data encoding the mapping between the hand pose space and the image feature space. The intrinsic dimension of such nonlinear mapping is learned by ISOSOM, which clusters the data into a lattice map. We perform 3D hand posture estimation on this map, showing that the ISOSOM algorithm performs better than traditional image retrieval algorithms for pose estimation. We also show that a 2.5D feature representation based on depth edges is clearly superior to intensity edge features commonly used in previous methods.

/pdf/the-isometric-self-organizing-map-for-3d-hand-pose-najsu8k0ds.pdf

The isometric self-organizing map for 3D hand pose estimation

This thesis describes a vision system which performs face recognition as a specialpurpose visual task, or "visual behavior". In addition to performing experiments using stored face images digitized under a range of imaging conditions, I have implemented face recognition in a near-real-time (or "interactive-time") computer system which locates and tracks a subject's head and then recognize the person by comparing characteristics of the face to those of known individuals. The computational approach of this system is motivated by both biology and information theory, as well as by the practical requirements of interactive-time performance and accuracy. The face recognition problem is treated as an intrinsically two-dimensional recognition problem, taking advantage of the fact that faces are normally upright and thus may be described by a small set of 2-D characteristic views. Each view is represented by a set of "eigenfaces" which are the significant eigenvectors (principal components) of the set of known faces. They form a holistic representation and do not necessarily correspond to individual features such as eyes, ears, and noses. This approach provides for the ability to learn and later recognize new faces in an unsupervised manner. In addition to face recognition, I explore other visual behaviors in the domain of human-computer interaction. Thesis Supervisor: Alex P. Pentland Associate Professor, MIT Media Laboratory

Interactive-time vision: face recognition as a visual behavior

This paper explores the use of fast, simple computer vision techniques to add compelling visual capabilities to social user interfaces. Social interfaces involve the user in natural dialog with animated, "lifelike" characters. However, current systems employ spoken language as the only input modality. Used effectively, vision can greatly enhance the user's experience interacting with these characters. In addition, vision can provide key information to help manage the dialog and to aid the speech recognition process. We describe constraints imposed by the conversational environment and present a set of "interactive-time" vision routines that begin to support the user's expectations of a seeing character. A control structure is presented which chooses among the vision routines based on the current state of the character, the conversation and the visual environment. These capabilities are beginning to be integrated into the Persona lifelike character project.

Visual interaction with lifelike characters

Various embodiments each include at least one of systems, methods, devices, and software for an augmented shared visual space for live mobile remote collaboration on physical tasks. One or more participants in location A can explore a scene in location B independently of one or more local participants current camera position in location B, and can communicate via spatial annotations that are immediately visible to all other participants in augmented reality.

Matthew Turk

Papers

Automatic Hot Spot Detection and Segmentation in Whole Body FDG-PET Images

The isometric self-organizing map for 3D hand pose estimation

Interactive-time vision: face recognition as a visual behavior

Visual interaction with lifelike characters

Systems and methods for augmented reality-based remote collaboration