Showing papers on "Expectation–maximization algorithm published in 2020"

Deep Neural Network for Robust Modulation Classification Under Uncertain Noise Conditions

Abstract: Recently, classifying the modulation schemes of signals using deep neural network has received much attention. In this paper, we introduce a general model of deep neural network (DNN)-based modulation classifiers for single-input single-output (SISO) systems. Its feasibility is analyzed using maximum a posteriori probability (MAP) criterion and its robustness to uncertain noise conditions is compared to that of the conventional maximum likelihood (ML)-based classifiers. To reduce the design and training cost of DNN classifiers, a simple but effective pre-processing method is introduced and adopted. Furthermore, featuring multiple recurrent neural network (RNN) layers, the DNN modulation classifier is realized. Simulation results show that the proposed RNN-based classifier is robust to the uncertain noise conditions, and the performance of it approaches to that of the ideal ML classifier with perfect channel and noise information. Moreover, with a much lower complexity, it outperforms the existing ML-based classifiers, specifically, expectation maximization (EM) and expectation conditional maximization (ECM) classifiers which iteratively estimate channel and noise parameters. In addition, the proposed classifier is shown to be invariant to the signal distortion such as frequency offset. Furthermore, the adopted pre-processing method is shown to accelerate the training process of our proposed classifier, thus reducing the training cost. Lastly, the computational complexity of our proposed classifier is analyzed and compared to other traditional ones, which further demonstrates its overall advantage.

A Novel EM Identification Method for Hammerstein Systems With Missing Output Data

Abstract: This article concerns a novel auxiliary-model-based expectation maximization (EM) estimation method for Hammerstein systems with data loss by extending the EM method to estimate models with multiple parameter vectors. The novel EM method relaxes the requirements on an autoregression model with one parameter vector, interactively maximizes the expectation over multiple parameter vectors in a more general model, and uses the output of an auxiliary model to substitute the missing outputs in the information vector in iteration processes. A numerical simulation is employed to demonstrate the effectiveness of the proposed novel EM method.

Clustering multivariate functional data in group-specific functional subspaces

Abstract: With the emergence of numerical sensors in many aspects of every- day life, there is an increasing need in analyzing multivariate functional data. This work focuses on the clustering of such functional data, in order to ease their modeling and understanding. To this end, a novel clustering technique for multivariate functional data is presented. This method is based on a func- tional latent mixture model which fits the data in group-specific functional subspaces through a multivariate functional principal component analysis. A family of parsimonious models is obtained by constraining model parameters within and between groups. An EM algorithm is proposed for model inference and the choice of hyper-parameters is addressed through model selection. Nu- merical experiments on simulated datasets highlight the good performance of the proposed methodology compared to existing works. This algorithm is then applied to the analysis of the pollution in French cities for one year.

An Alternative to EM for Gaussian Mixture Models: Batch and Stochastic Riemannian Optimization

Abstract: We consider maximum likelihood estimation for Gaussian Mixture Models (Gmm s). This task is almost invariably solved (in theory and practice) via the Expectation Maximization (EM) algorithm. EM owes its success to various factors, of which is its ability to fulfill positive definiteness constraints in closed form is of key importance. We propose an alternative to EM grounded in the Riemannian geometry of positive definite matrices, using which we cast Gmm parameter estimation as a Riemannian optimization problem. Surprisingly, such an out-of-the-box Riemannian formulation completely fails and proves much inferior to EM. This motivates us to take a closer look at the problem geometry, and derive a better formulation that is much more amenable to Riemannian optimization. We then develop Riemannian batch and stochastic gradient algorithms that outperform EM, often substantially. We provide a non-asymptotic convergence analysis for our stochastic method, which is also the first (to our knowledge) such global analysis for Riemannian stochastic gradient. Numerous empirical results are included to demonstrate the effectiveness of our methods.

An improved stochastic EM algorithm for large-scale full-information item factor analysis

Abstract: In this paper, we explore the use of the stochastic EM algorithm (Celeux & Diebolt (1985) Computational Statistics Quarterly, 2, 73) for large-scale full-information item factor analysis. Innovations have been made on its implementation, including an adaptive-rejection-based Gibbs sampler for the stochastic E step, a proximal gradient descent algorithm for the optimization in the M step, and diagnostic procedures for determining the burn-in size and the stopping of the algorithm. These developments are based on the theoretical results of Nielsen (2000, Bernoulli, 6, 457), as well as advanced sampling and optimization techniques. The proposed algorithm is computationally efficient and virtually tuning-free, making it scalable to large-scale data with many latent traits (e.g. more than five latent traits) and easy to use for practitioners. Standard errors of parameter estimation are also obtained based on the missing-information identity (Louis, 1982, Journal of the Royal Statistical Society, Series B, 44, 226). The performance of the algorithm is evaluated through simulation studies and an application to the analysis of the IPIP-NEO personality inventory. Extensions of the proposed algorithm to other latent variable models are discussed.

Robust Generalized Point Cloud Registration With Orientational Data Based on Expectation Maximization

Abstract: This paper introduces a robust generalized point cloud (PC) registration method that utilizes not only the positional but also the orientation information associated with each point. The proposed method solves the rigid PC registration problem in a probabilistic manner, which casts the problem into a maximum likelihood (ML) framework. A hybrid mixture model (HMM) is utilized to represent one generalized PC. In the HMM, a von-Mises–Fisher mixture model (FMM) is adopted to model the orientational uncertainty, while a Gaussian mixture model (GMM) is used to represent the positional uncertainty. An expectation–maximization (EM) algorithm is adopted to solve the optimization problem in an iterative manner to find the optimal rotation matrix and the translation vector between two generalized PCs. In both expectation step (E step) and maximization step (M step), orientational information is utilized, which can potentially improve the algorithm’s robustness to noise and outliers. In the E step, the posterior probabilities that represent the degree of point correspondences in two PCs are computed. In the M step, an efficient closed-form solution to a rigid transformation matrix is developed. E and M steps will iterate until certain convergence criteria are satisfied. Extensive experiments under different noise levels and outlier ratios have been carried out on a data set of femur bone computed tomography images. Experimental results show that the proposed method outperforms the state-of-the-art ones in terms of accuracy, robustness, and convergence speed significantly. Note to Practitioners —This paper was motivated by solving the problem of registering two PCs. Most existing approaches generally use only the positional information associated with each point and thus lack robustness to noise and outliers. This paper suggests a new robust method that also adopts the normal vectors associated with each point. The registration problem is cast into a maximum likelihood (ML) problem and solved under the expectation–maximization (EM) framework. Closed-form solutions for estimating parameters in both expectation and maximization steps are provided in this paper. We have demonstrated through extensive experiments that the proposed registration algorithm achieves improved accuracy, robustness to noise and outliers, and faster convergence speed.

Prognostics for Linear Stochastic Degrading Systems With Survival Measurements

Abstract: This paper addresses the problem of how to estimate the remaining useful life (RUL) for partially observed linear stochastic degrading systems with survival measurements. The motivation of this paper arises from two engineering facts—the measured degradation signals are taken from a survival degradation path and the degradation progression of the system is inevitably affected by multisource variability. To do so, we first revisit the linear degradation modeling framework driven by the Wiener process to account for the above two facts, and a new state-space model with non-Gaussian state transitions is constructed based on the constraint of survival measurements. Then, the particle filtering algorithm is applied to real-time estimate the non-Gaussian degradation state and random-effect parameter from measurements. Furthermore, we derive the probabilistic distribution of the RUL which can be real-time updated based on the available measurements. To apply the presented method, a two-stage estimation procedure based on particle expectation maximization algorithm is presented to determine and update the model parameters. The novelty of the presented method is to exclude the probability of failure before the current monitoring time and account for both impacts of the multisource variability and surviving degradation path on the RUL estimation. Finally, we demonstrate the proposed approach by a case study on gyros in the inertial platform.

DirectPET: full-size neural network PET reconstruction from sinogram data.

Abstract: Purpose: Neural network image reconstruction directly from measurement data is a relatively new field of research, which until now has been limited to producing small single-slice images (e.g., 1 × 128 × 128). We proposed a more efficient network design for positron emission tomography called DirectPET, which is capable of reconstructing multislice image volumes (i.e., 16 × 400 × 400) from sinograms. Approach: Large-scale direct neural network reconstruction is accomplished by addressing the associated memory space challenge through the introduction of a specially designed Radon inversion layer. Using patient data, we compare the proposed method to the benchmark ordered subsets expectation maximization (OSEM) algorithm using signal-to-noise ratio, bias, mean absolute error, and structural similarity measures. In addition, line profiles and full-width half-maximum measurements are provided for a sample of lesions. Results: DirectPET is shown capable of producing images that are quantitatively and qualitatively similar to the OSEM target images in a fraction of the time. We also report on an experiment where DirectPET is trained to map low-count raw data to normal count target images, demonstrating the method’s ability to maintain image quality under a low-dose scenario. Conclusion: The ability of DirectPET to quickly reconstruct high-quality, multislice image volumes suggests potential clinical viability of the method. However, design parameters and performance boundaries need to be fully established before adoption can be considered.

Unsupervised Linear and Nonlinear Channel Equalization and Decoding Using Variational Autoencoders

Abstract: A new approach for blind channel equalization and decoding, variational inference, and variational autoencoders (VAEs) in particular, is introduced. We first consider the reconstruction of uncoded data symbols transmitted over a noisy linear intersymbol interference (ISI) channel, with an unknown impulse response, without using pilot symbols. We derive an approximate maximum likelihood estimate to the channel parameters and reconstruct the transmitted data. We demonstrate significant and consistent improvements in the error rate of the reconstructed symbols, compared to existing blind equalization methods such as constant modulus, thus enabling faster channel acquisition. The VAE equalizer uses a convolutional neural network with a small number of free parameters. These results are extended to blind equalization over a noisy nonlinear ISI channel with unknown parameters. We then consider coded communication using low-density parity-check (LDPC) codes transmitted over a noisy linear or nonlinear ISI channel. The goal is to reconstruct the transmitted message from the channel observations corresponding to a transmitted codeword, without using pilot symbols. We demonstrate improvements compared to the expectation maximization (EM) algorithm using turbo equalization. Furthermore, unlike EM, the computational complexity of our method does not have exponential dependence on the size of the channel impulse response.

Gaussian parsimonious clustering models with covariates and a noise component

Abstract: We consider model-based clustering methods for continuous, correlated data that account for external information available in the presence of mixed-type fixed covariates by proposing the MoEClust suite of models. These models allow different subsets of covariates to influence the component weights and/or component densities by modelling the parameters of the mixture as functions of the covariates. A familiar range of constrained eigen-decomposition parameterisations of the component covariance matrices are also accommodated. This paper thus addresses the equivalent aims of including covariates in Gaussian parsimonious clustering models and incorporating parsimonious covariance structures into all special cases of the Gaussian mixture of experts framework. The MoEClust models demonstrate significant improvement from both perspectives in applications to both univariate and multivariate data sets. Novel extensions to include a uniform noise component for capturing outliers and to address initialisation of the EM algorithm, model selection, and the visualisation of results are also proposed.

Multi-class ECG beat classification based on a Gaussian mixture model of Karhunen-Loève transform

Abstract: Cardiovascular diseases are one of the main causes of death around the world. Automatic classification of electrocardiogram (ECG) signals is of paramount importance in the unmanned detection of a wide range of heartbeat abnormalities. In this paper an effective multi-class beat classifier, based on a statistical identification of a minimum-complexity model, is presented. This methodology extracts from the ECG signal the multivariate relationships of its natural modes, by means of the separation property of the Karhunen-Loeve transform (KLT). Then, it exploits an optimized expectation maximization (EM) algorithm to find the optimal parameters of a Gaussian mixture model, with the focus being in reducing the number of parameters. The resulting statistical model is thus based on the estimation of the multivariate probability density function (PDF) that characterizes each beat type. Based on the above statistical characterization a multi-class ECG classification was performed. The experiments, conducted on the ECG signals from the MIT-BIH arrhythmia database, demonstrated the validity and, considering the reduced model size, the excellent performance of this technique to classify the ECG signals into different disease categories.

Robust Extreme Learning Machine for Modeling with Unknown Noise

Abstract: Extreme learning machine (ELM) is an emerging machine learning technique for training single hidden layer feedforward networks (SLFNs). During the training phase, ELM model can be created by simultaneously minimizing the modeling errors and norm of the output weights. Usually, squared loss is widely utilized in the objective function of ELMs, which is theoretically optimal for the Gaussian error distribution. However, in practice, data collected from uncertain and heterogeneous environments trivially result in unknown noise, which may be very complex and cannot be described well using any single distribution. In order to tackle this issue, in this paper, a robust ELM (R-ELM) is proposed for improving the modeling capability and robustness with Gaussian and non-Gaussian noise. In R-ELM, a modified objective function is constructed to fit the noise using mixture of Gaussian (MoG) to approximate any continuous distribution. In addition, the corresponding solution for the new objective function is developed based on expectation maximization (EM) algorithm. Comprehensive experiments, both on selected benchmark datasets and real world applications, demonstrate that the proposed R-ELM has better robustness and generalization performance than state-of-the-art machine learning approaches.

Imputation and low-rank estimation with Missing Non At Random data

Abstract: Missing values challenge data analysis because many supervised and unsu-pervised learning methods cannot be applied directly to incomplete data. Matrix completion based on low-rank assumptions are very powerful solution for dealing with missing values. However, existing methods do not consider the case of informative missing values which are widely encountered in practice. This paper proposes matrix completion methods to recover Missing Not At Random (MNAR) data. Our first contribution is to suggest a model-based estimation strategy by modelling the missing mechanism distribution. An EM algorithm is then implemented, involving a Fast Iterative Soft-Thresholding Algorithm (FISTA). Our second contribution is to suggest a computationally efficient surrogate estimation by implicitly taking into account the joint distribution of the data and the missing mechanism: the data matrix is concatenated with the mask coding for the missing values ; a low-rank structure for exponential family is assumed on this new matrix, in order to encode links between variables and missing mechanisms. The methodology that has the great advantage of handling different missing value mechanisms is robust to model specification errors.

Improved Initialization of the EM Algorithm for Mixture Model Parameter Estimation

Abstract: A commonly used tool for estimating the parameters of a mixture model is the Expectation–Maximization (EM) algorithm, which is an iterative procedure that can serve as a maximum-likelihood estimator. The EM algorithm has well-documented drawbacks, such as the need for good initial values and the possibility of being trapped in local optima. Nevertheless, because of its appealing properties, EM plays an important role in estimating the parameters of mixture models. To overcome these initialization problems with EM, in this paper, we propose the Rough-Enhanced-Bayes mixture estimation (REBMIX) algorithm as a more effective initialization algorithm. Three different strategies are derived for dealing with the unknown number of components in the mixture model. These strategies are thoroughly tested on artificial datasets, density–estimation datasets and image–segmentation problems and compared with state-of-the-art initialization methods for the EM. Our proposal shows promising results in terms of clustering and density-estimation performance as well as in terms of computational efficiency. All the improvements are implemented in the rebmix R package.

A Unified Probabilistic Monitoring Framework for Multimode Processes Based on Probabilistic Linear Discriminant Analysis

Abstract: This article develops a novel probabilistic monitoring framework for industrial processes with multiple operational conditions. The proposed method is based on the probabilistic linear discriminant analysis (PLDA), which relies on two sets of latent variables, i.e., the between-class and within-class latent variables. In order to deal with the large within-class variations in multi-mode industrial processes, this approach modifies the original PLDA by introducing a separate within-class loading matrix for each operational mode and designs an expectation maximization (EM) algorithm to estimate the model parameters from the training samples. Mode identification for test samples is achieved by investigating the cosine similarity in the between-class latent variables and two monitoring statistics corresponding to within-class latent variables and the residuals are considered for fault detection. To diagnose the process fault, this article further develops a sparse probabilistic generative model based on PLDA for fault isolation. The enhanced performance of the proposed method is illustrated by applications to numerical examples and industrial processes.

Improved Most Likely Heteroscedastic Gaussian Process Regression via Bayesian Residual Moment Estimator

Abstract: This paper proposes an improved most likely heteroscedastic Gaussian process (MLHGP) algorithm to handle a kind of nonlinear regression problems involving input-dependent noise. The improved MLHGP follows the same learning scheme as the current algorithm by use of two Gaussian processes (GPs), with the first GP for recovering the unknown function and the second GP for modeling the input-dependent noise. Unlike the current MLHGP pursuing an empirical estimate of the noise level which is provably biased in most of local noise cases, the improved algorithm gives rise to an approximately unbiased estimate of the input-dependent noise. The approximately unbiased noise estimate is elicited from Bayesian residuals by the method of moments. As a by-product of this improvement, the expectation maximization (EM)-like procedure in the current MLHGP is avoided such that the improved algorithm requires only standard GP learnings to be performed twice. Four benchmark experiments, consisting of two synthetic cases and two real-world datasets, demonstrate that the improved MLHGP algorithm outperforms the current version not only in accuracy and stability, but also in computational efficiency.

Learning the Clustering of Longitudinal Shape Data Sets into a Mixture of Independent or Branching Trajectories

Abstract: Given repeated observations of several subjects over time, i.e. a longitudinal data set, this paper introduces a new model to learn a classification of the shapes progression in an unsupervised setting: we automatically cluster a longitudinal data set in different classes without labels. Our method learns for each cluster an average shape trajectory (or representative curve) and its variance in space and time. Representative trajectories are built as the combination of pieces of curves. This mixture model is flexible enough to handle independent trajectories for each cluster as well as fork and merge scenarios. The estimation of such non linear mixture models in high dimension is known to be difficult because of the trapping states effect that hampers the optimisation of cluster assignments during training. We address this issue by using a tempered version of the stochastic EM algorithm. Finally, we apply our algorithm on different data sets. First, synthetic data are used to show that a tempered scheme achieves better convergence. We then apply our method to different real data sets: 1D RECIST score used to monitor tumors growth, 3D facial expressions and meshes of the hippocampus. In particular, we show how the method can be used to test different scenarios of hip-pocampus atrophy in ageing by using an heteregenous population of normal ageing individuals and mild cog-nitive impaired subjects.

Robust Factorization Methods Using a Gaussian/Uniform Mixture Model.

Abstract: In this paper we address the problem of building a class of robust factorization algorithms that solve for the shape and motion parameters with both affine (weak perspective) and perspective camera models. We introduce a Gaussian/uniform mixture model and its associated EM algorithm. This allows us to address robust parameter estimation within a data clustering approach. We propose a robust technique that works with any affine factorization method and makes it robust to outliers. In addition, we show how such a framework can be further embedded into an iterative perspective factorization scheme. We carry out a large number of experiments to validate our algorithms and to compare them with existing ones. We also compare our approach with factorization methods that use M-estimators.

Characterizing photon number statistics using conjugate optical homodyne detection.

Abstract: We study the problem of determining the photon number statistics of an unknown quantum state using conjugate optical homodyne detection. We quantify the information gain in a single-shot measurement and show that the photon number statistics can be recovered in repeated measurements on an ensemble of identical input states without scanning the phase of the input state or randomizing the phase of the local oscillator used in homodyne detection. We demonstrate how the expectation maximization algorithm and Bayesian inference can be utilized to facilitate the reconstruction and illustrate our approach by conducting experiments to study the photon number distributions of a weak coherent state and a thermal state source.

Breast Cancer Detection With Low-Dimensional Ordered Orthogonal Projection in Terahertz Imaging

Abstract: This article proposes a new dimension reduction algorithm based on low-dimensional ordered orthogonal projection, which is used for cancer detection with terahertz (THz) images of freshly excised human breast cancer tissues. A THz image can be represented by a data cube with each pixel containing a high-dimensional spectrum vector covering several THz frequencies, where each frequency represents a different dimension in the vector. The proposed algorithm projects the high-dimensional spectrum vector of each pixel within the THz image into a low-dimensional subspace that contains the majority of the unique features embedded in the image. The low-dimensional subspace is constructed by sequentially identifying its orthonormal basis vectors, such that each newly chosen basis vector represents the most unique information not contained by existing basis vectors. A multivariate Gaussian mixture model is used to represent the statistical distributions of the low-dimensional feature vectors obtained from the proposed dimension reduction algorithm. The model parameters are iteratively learned by using unsupervised learning methods, such as Markov chain Monte Carlo or expectation maximization, and the results are used to classify the various regions within a tumor sample. Experiment results demonstrate that the proposed method achieves apparent performance improvement in human breast cancer tissue over existing approaches such as one-dimensional Markov chain Monte Carlo. The results confirm that the dimension reduction algorithm presented in this article is a promising technique for breast cancer detection with THz images, and the classification results present a good correlation with respect to the histopathology results of the analyzed samples.

Diagnostic Analytics for an Autoregressive Model under the Skew-Normal Distribution

Abstract: Autoregressive models have played an important role in time series. In this paper, an autoregressive model based on the skew-normal distribution is considered. The estimation of its parameters is carried out by using the expectation–maximization algorithm, whereas the diagnostic analytics are conducted by means of the local influence method. Normal curvatures for the model under four perturbation schemes are established. Simulation studies are conducted to evaluate the performance of the proposed procedure. In addition, an empirical example involving weekly financial return data are analyzed using the procedure with the proposed diagnostic analytics, which has improved the model fit.

Distribution Optimization: An evolutionary algorithm to separate Gaussian mixtures

Abstract: Finding subgroups in biomedical data is a key task in biomedical research and precision medicine. Already one-dimensional data, such as many different readouts from cell experiments, preclinical or human laboratory experiments or clinical signs, often reveal a more complex distribution than a single mode. Gaussian mixtures play an important role in the multimodal distribution of one-dimensional data. However, although fitting of Gaussian mixture models (GMM) is often aimed at obtaining the separate modes composing the mixture, current technical implementations, often using the Expectation Maximization (EM) algorithm, are not optimized for this task. This occasionally results in poorly separated modes that are unsuitable for determining a distinguishable group structure in the data. Here, we introduce "Distribution Optimization" an evolutionary algorithm to GMM fitting that uses an adjustable error function that is based on chi-square statistics and the probability density. The algorithm can be directly targeted at the separation of the modes of the mixture by employing additional criterion for the degree by which single modes overlap. The obtained GMM fits were comparable with those obtained with classical EM based fits, except for data sets where the EM algorithm produced unsatisfactory results with overlapping Gaussian modes. There, the proposed algorithm successfully separated the modes, providing a basis for meaningful group separation while fitting the data satisfactorily. Through its optimization toward mode separation, the evolutionary algorithm proofed particularly suitable basis for group separation in multimodally distributed data, outperforming alternative EM based methods.

Skeletonization of Plant Point Cloud Data Using Stochastic Optimization Framework.

Abstract: Skeleton extraction from 3D plant point cloud data is an essential prior for myriads of phenotyping studies. Although skeleton extraction from 3D shapes have been studied extensively in the computer vision and graphics literature, handling the case of plants is still an open problem. Drawbacks of the existing approaches include the zigzag structure of the skeleton, nonuniform density of skeleton points, lack of points in the areas having complex geometry structure, and most importantly the lack of biological relevance. With the aim to improve existing skeleton structures of state-of-the-art, we propose a stochastic framework which is supported by the biological structure of the original plant (we consider plants without any leaves). Initially we estimate the branching structure of the plant by the notion of beta-splines to form a curve tree defined as a finite set of curves joined in a tree topology with certain level of smoothness. In the next phase, we force the discrete points in the curve tree to move towards the original point cloud by treating each point in the curve tree as a center of Gaussian, and points in the input cloud data as observations from the Gaussians. The task is to find the correct locations of the Gaussian centroids by maximizing a likelihood. The optimization technique is iterative and is based on the Expectation Maximization (EM) algorithm. The E-step estimates which Gaussian the observed point cloud was sampled from, and the M-step maximizes the negative log-likelihood that the observed points were sampled from the Gaussian Mixture Model (GMM) with respect to the model parameters. We experiment with several real world and synthetic datasets and demonstrate the robustness of the approach over the state-of-the-art.

Mini-batch learning of exponential family finite mixture models

Abstract: Mini-batch algorithms have become increasingly popular due to the requirement for solving optimization problems, based on large-scale data sets. Using an existing online expectation–maximization (EM) algorithm framework, we demonstrate how mini-batch (MB) algorithms may be constructed, and propose a scheme for the stochastic stabilization of the constructed mini-batch algorithms. Theoretical results regarding the convergence of the mini-batch EM algorithms are presented. We then demonstrate how the mini-batch framework may be applied to conduct maximum likelihood (ML) estimation of mixtures of exponential family distributions, with emphasis on ML estimation for mixtures of normal distributions. Via a simulation study, we demonstrate that the mini-batch algorithm for mixtures of normal distributions can outperform the standard EM algorithm. Further evidence of the performance of the mini-batch framework is provided via an application to the famous MNIST data set.