Many problems in the sciences and engineering can be rephrased as optimization problems on matrix search spaces endowed with a so-called manifold structure. This book shows how to exploit the special structure of such problems to develop efficient numerical algorithms. It places careful emphasis on both the numerical formulation of the algorithm and its differential geometric abstraction--illustrating how good algorithms draw equally from the insights of differential geometry, optimization, and numerical analysis. Two more theoretical chapters provide readers with the background in differential geometry necessary to algorithmic development. In the other chapters, several well-known optimization methods such as steepest descent and conjugate gradients are generalized to abstract manifolds. The book provides a generic development of each of these methods, building upon the material of the geometric chapters. It then guides readers through the calculations that turn these geometrically formulated methods into concrete numerical algorithms. The state-of-the-art algorithms given as examples are competitive with the best existing algorithms for a selection of eigenspace problems in numerical linear algebra. Optimization Algorithms on Matrix Manifolds offers techniques with broad applications in linear algebra, signal processing, data mining, computer vision, and statistical analysis. It can serve as a graduate-level textbook and will be of interest to applied mathematicians, engineers, and computer scientists.

http://www.eeci-institute.eu/GSC2011/Photos-EECI/EECI-GSC-2011-M5/book_AMS.pdf

Optimization Algorithms on Matrix Manifolds

Independent Component Analysis.

Edited by the people who were forerunners in creating the field, together with contributions from 34 leading international experts, this handbook provides the definitive reference on Blind Source Separation, giving a broad and comprehensive description of all the core principles and methods, numerical algorithms and major applications in the fields of telecommunications, biomedical engineering and audio, acoustic and speech processing. Going beyond a machine learning perspective, the book reflects recent results in signal processing and numerical analysis, and includes topics such as optimization criteria, mathematical tools, the design of numerical algorithms, convolutive mixtures, and time frequency approaches. This Handbook is an ideal reference for university researchers, RD algebraic identification of under-determined mixtures, time-frequency methods, Bayesian approaches, blind identification under non negativity approaches, semi-blind methods for communicationsShows the applications of the methods to key application areas such as telecommunications, biomedical engineering, speech, acoustic, audio and music processing, while also giving a general method for developing applications

https://ie.technion.ac.il/~mcib/Gribonval_Zibulevsky_SCA_chapter.pdf

Handbook of Blind Source Separation: Independent Component Analysis and Applications

Tensors or multiway arrays are functions of three or more indices $(i,j,k,\ldots)$ —similar to matrices (two-way arrays), which are functions of two indices $(r,c)$ for (row, column). Tensors have a rich history, stretching over almost a century, and touching upon numerous disciplines; but they have only recently become ubiquitous in signal and data analytics at the confluence of signal processing, statistics, data mining, and machine learning. This overview article aims to provide a good starting point for researchers and practitioners interested in learning about and working with tensors. As such, it focuses on fundamentals and motivation (using various application examples), aiming to strike an appropriate balance of breadth and depth that will enable someone having taken first graduate courses in matrix algebra and probability to get started doing research and/or developing tensor algorithms and software. Some background in applied optimization is useful but not strictly required. The material covered includes tensor rank and rank decomposition; basic tensor factorization models and their relationships and properties (including fairly good coverage of identifiability); broad coverage of algorithms ranging from alternating optimization to stochastic gradient; statistical performance analysis; and applications ranging from source separation to collaborative filtering, mixture and topic modeling, classification, and multilinear subspace learning.

Tensor Decomposition for Signal Processing and Machine Learning

The Foundations of Statistics By Prof. Leonard J. Savage. (Wiley Publications in Statistics.) Pp. xv + 294. (New York; John Wiley and Sons, Inc.; London: Chapman and Hall, Ltd., 1954.) 48s. net.

Probability and Statistical Inference

Approximate joint diagonalization of a set of matrices is an essential tool in many blind source separation (BSS) algorithms. A common measure of the attained diagonalization of the set is the weighted least-squares (WLS) criterion. However, most well-known algorithms are restricted to finding an orthogonal diagonalizing matrix, relying on a whitening phase for the nonorthogonal factor. Often, such an approach implies unbalanced weighting, which can result in degraded performance. We propose an iterative alternating-directions algorithm for minimizing the WLS criterion with respect to a general (not necessarily orthogonal) diagonalizing matrix. Under some mild assumptions, we prove weak convergence in the sense that the norm of parameters update is guaranteed to fall below any arbitrarily small threshold within a finite number of iterations. We distinguish between Hermitian and symmetrical problems. Using BSS simulations results, we demonstrate the improvement in estimating the mixing matrix, resulting from the relaxation of the orthogonality restriction.

/pdf/non-orthogonal-joint-diagonalization-in-the-least-squares-269m9bogvq.pdf

Non-orthogonal joint diagonalization in the least-squares sense with application in blind source separation

An apparatus and method for the analysis of a sequence of captured images covering a scene for detecting and tracking of moving and static objects (86) and for matching (88) the patterns of object behavior in the captured images to object behavior in predetermined scenarios.

Method and apparatus for video frame sequence-based object tracking

We propose a new low-complexity approximate joint diagonalization (AJD) algorithm, which incorporates nontrivial block-diagonal weight matrices into a weighted least-squares (WLS) AJD criterion. Often in blind source separation (BSS), when the sources are nearly separated, the optimal weight matrix for WLS-based AJD takes a (nearly) block-diagonal form. Based on this observation, we show how the new algorithm can be utilized in an iteratively reweighted separation scheme, thereby giving rise to fast implementation of asymptotically optimal BSS algorithms in various scenarios. In particular, we consider three specific (yet common) scenarios, involving stationary or block-stationary Gaussian sources, for which the optimal weight matrices can be readily estimated from the sample covariance matrices (which are also the target-matrices for the AJD). Comparative simulation results demonstrate the advantages in both speed and accuracy, as well as compliance with the theoretically predicted asymptotic optimality of the resulting BSS algorithms based on the weighted AJD, both on large scale problems with matrices of the size 100times100.

/pdf/fast-approximate-joint-diagonalization-incorporating-weight-2pxhemi753.pdf

Fast Approximate Joint Diagonalization Incorporating Weight Matrices

We present a dictionary attack that is based on keyboard acoustic emanations. We combine signal processing and efficient data structures and algorithms, to successfully reconstruct single words of 7-13 characters from a recording of the clicks made when typing them on a keyboard. Our attack does not require any training, and works on an individual recording of the typed word (may be under 5 seconds of sound). The attack is very efficient, taking under 20 seconds per word on a standard PC. We demonstrate a 90% or better success rate of finding the correct word in the top 50 candidates identified by the attack, for words of 10 or more characters, and a success rate of 73% over all the words we tested. We show that the dominant factors affecting the attack's success are the word length, and more importantly, the number of repeated characters within the word. Our attack can be used as an effective acoustic-based password cracker. Our attack can also be used as part of an acoustic long-text reconstruction method, that is much more efficient and requires much less text than previous approaches.

/pdf/dictionary-attacks-using-keyboard-acoustic-emanations-1jwqd4h54s.pdf

Dictionary attacks using keyboard acoustic emanations

Blind separation of Gaussian sources with different spectra can be attained using second-order statistics. The second-order blind identification (SOBI) algorithm, proposed by Belouchrani et al. (1997), uses approximate joint diagonalization. We show that substantial improvement over SOBI can be attained when the joint diagonalization is transformed into a properly weighted nonlinear least squares problem. We provide an iterative solution and derive the optimal weights for our weights-adjusted SOBI (WASOBI) algorithm. The improvement is demonstrated by analysis and simulations.

/pdf/blind-separation-of-gaussian-sources-via-second-order-2x10z7dyyz.pdf

Arie Yeredor

Papers

Non-orthogonal joint diagonalization in the least-squares sense with application in blind source separation

Method and apparatus for video frame sequence-based object tracking

Fast Approximate Joint Diagonalization Incorporating Weight Matrices

Dictionary attacks using keyboard acoustic emanations

Blind separation of Gaussian sources via second-order statistics with asymptotically optimal weighting