Showing papers on "Expectation–maximization algorithm published in 1987"

The calculation of posterior distributions by data augmentation

Abstract: The idea of data augmentation arises naturally in missing value problems, as exemplified by the standard ways of filling in missing cells in balanced two-way tables. Thus data augmentation refers to a scheme of augmenting the observed data so as to make it more easy to analyze. This device is used to great advantage by the EM algorithm (Dempster, Laird, and Rubin 1977) in solving maximum likelihood problems. In situations when the likelihood cannot be approximated closely by the normal likelihood, maximum likelihood estimates and the associated standard errors cannot be relied upon to make valid inferential statements. From the Bayesian point of view, one must now calculate the posterior distribution of parameters of interest. If data augmentation can be used in the calculation of the maximum likelihood estimate, then in the same cases one ought to be able to use it in the computation of the posterior distribution. It is the purpose of this article to explain how this can be done. The basic idea ...

On Bootstrapping the Likelihood Ratio Test Statistic for the Number of Components in a Normal Mixture

Abstract: An important but difficult problem in practice is assessing the number of components g in a mixture. An obvious way of proceeding is to use the likelihood ratio test statistic λ to test for the smallest value of g consistent with the data. Unfortunately with mixture models, regularity conditions do not hold for –2 log λ to have it usual asymptotic null distribution of chi‐squared. In this paper the role of the bootstrap is highlighted for the assessment of the null distribution of –2 log λ for the test of a single normal density versus a mixture of two normal densities in the univariate case.

A Maximum a Posteriori Probability Expectation Maximization Algorithm for Image Reconstruction in Emission Tomography

Abstract: The expectation maximization method for maximum likelihood image reconstruction in emission tomography, based on the Poisson distribution of the statistically independent components of the image and measurement vectors, is extended to a maximum aposteriori image reconstruction using a multivariate Gaussian a priori probability distribution of the image vector. The approach is equivalent to a penalized maximum likelihood estimation with a special choice of the penalty function. The expectation maximization method is applied to find the a posteriori probability maximizer. A simple iterative formula is derived for a penalty function that is a weighted sum of the squared deviations of image vector components from their a priori mean values. The method is demonstrated to be superior to pure likelihood maximization, in that the penalty function prevents the occurrence of irregular high amplitude patterns in the image with a large number of iterations (the so-called "checkerboard effect" or "noise artifact").

Maximum likelihood computations with repeated measures: application of the EM algorithm

Abstract: The purpose of this article is to consider the use of the EM algorithm (Dempster, Laird, and Rubin 1977) for both maximum likelihood (ML) and restricted maximum likelihood (REML) estimation in a general repeated measures setting using a multivariate normal data model with linear mean and covariance structure (Anderson 1973). Several models and methods of analysis have been proposed in recent years for repeated measures data; Ware (1985) presented an overview. Because the EM algorithm is a general-purpose, iterative method for computing ML estimates with incomplete data, it has often been used in this particular setting (Dempster et al. 1977; Andrade and Helms 1984; Jennrich and Schluchter 1985). There are two apparently different approaches to using the EM algorithm in this setting. In one application, each experimental unit is observed under a standard protocol specifying measurements at each of n occasions (or under n conditions), and incompleteness implies that the number of measurements actua...

Implementing and Accelerating the EM Algorithm for Positron Emission Tomography

Abstract: Since the publication of Shepp and Vadi's [ 14] maximum likelihood reconstruction algorithm for emission tomography (ET), many medical research centers engaged in ET have made an effort to change their reconstruction algorithms to this new approach. Some have succeeded, while others claim they could not adopt this new approach primarily because of limited computing power. In this paper, we discuss techniques for reducing the computational requirements of the reconstruction algorithm. Specifically, the paper discusses the data structures one might use and ways of taking advantage of the geometry of the physical system. The paper also treats some of the numerical aspects of the EM (expectation maximization) algorithm, and ways of speeding up the numerical algorithm using some of the traditional techniques of numerical analysis.

Maximum likelihood estimation from incomplete data

Abstract: SUMMARY Y is a linear regression on a variable X; X is fixed and all its sample values are observed. Y, on the other hand, has some sample values missing. This work outlines a maximum likelihood (ml) procedure that tries to adjust for bias due to non-random missingness; here non-randomness is specified by a logistic distribution. The ml procedure is implemented via two iterative technologies, namely the EM algorithm (of Dempster, Laird & Rubin, 1977) and the Newton-Raphson method. Data from a dialysis study are used to illustrate our estimation procedure, and results show that the ml procedure is quite effective in adjusting for bias.

Maximum Likelihood Estimation of Mixed Stock Fishery Composition

Abstract: In this paper the maximum likelihood method of estimating stock composition is initially presented under the assumption that all of the sampled characteristics are of discrete type (electrophoretic...

Covariate measurement error in generalized linear models

Abstract: SUMMARY The EM algorithm is used to obtain estimators of regression coefficients for generalized linear models with canonical link when normally distributed covariates are masked by normally distributed measurement errors. By casting the true covariates as 'missing data', the EM procedure suggests an iterative scheme in which each cycle consists of an E-step, requiring the computation of approximate first and second conditional moments of the true covariates given the observed data, followed by an M-step in which regression parameters are updated by iteratively reweighted least squares based on these approximations. The proposed procedure is compared numerically with the exact maximum likelihood solution, obtained by using Gaussian quadrature instead of the approximations in the E-step of the EM algorithm, and with alternative estimators for simple logistic regression with measurement error. The results for the proposed procedure are encouraging. A procedure is proposed in this paper for estimating regression coefficients in generalized linear models when one or more of the covariates is measured with error. This work is related to the paper by Carroll et al. (1984) in which maximum likelihood estimates are considered for the structural logistic regression model. These authors provide estimates for the computationally simpler probit regression model and suggest that the structural logistic regression can be done in principle. This paper follows up on that suggestion but differs in detail from similar work by Armstrong (1985). Other procedures with related aims have been suggested by Stefanski & Carroll (1985), Wolter & Fuller (1982) and Prentice (1982). For a quick introduction to the proposed method it is convenient to display the naive estimator which ignores measurement error in its standard computational form. If yi is an observed response variable and xi is the observed covariate, then, for the model which equates the canonical parameter of the distribution of yi to xtf3, the maximum likelihood estimator of f3 can be obtained by iteratively reweighted least-squares. The estimate of f3 after (s +1) cycles is given by

Maximum likelihood estimation of multivariate polyserial and polychoric correlation coefficients

Abstract: The method of finding the maximum likelihood estimates of the parameters in a multivariate normal model with some of the component variables observable only in polytomous form is developed. The main stratagem used is a reparameterization which converts the corresponding log likelihood function to an easily handled one. The maximum likelihood estimates are found by a Fletcher-Powell algorithm, and their standard error estimates are obtained from the information matrix. When the dimension of the random vector observable only in polytomous form is large, obtaining the maximum likelihood estimates is computationally rather labor expensive. Therefore, a more efficient method, the partition maximum likelihood method, is proposed. These estimation methods are demonstrated by real and simulated data, and are compared by means of a simulation study.

A Fast Reconstruction Algorthm for Stationary Positron Emission Tomography Based on a Modified EM Algorithm

Abstract: An efficient iterative reconstruction method for positron emission tomography (PET) is presented. The algorithm is basically an enhanced EM (expectation maximization) algorithm with improved frequency response. High-frequency components of the ratio of measured to calculated projections are extracted and are taken into account for the iterative correction of image density in such a way that the correction is performed with a uniform efficiency over the image plane and with a flat frequency response. As a result, the convergence speed is not so sensitive to the image pattern or matrix size as the standard EM algorithm, and nonuniformity of the spatial resolution is significantly improved. Nonnegativity of the reconstructed image is preserved. Simulation studies have been made assuming two PET systems: a scanning PET with ideal sampling and a stationary PET with sparse sampling. In the latter, a "bank array" of detectors is employed to improve the sampling in the object plane. The new algorithm provides satisfactory images by two or three iterations starting from a flat image in either case. The behavior of convergence is monitored by evaluating the root mean square of C(b)-1 where C(b) is the correction factor for pixel b in the EM algorithm. The value decreases rapidly and monotonically with iteration number. Although the theory is not accurate enough to assure the stability of convergence, the algorithm is promising to achieve significant saving in computation compared to the standard EM algorithm.

A fast scoring algorithm for maximum likelihood estimation in unbalanced mixed models with nested random effects

Abstract: SUMMARY A fast Fisher scoring algorithm for maximum likelihood estimation in unbalanced mixed models with nested random effects is described. The algorithm uses explicit formulae for the inverse and the determinant of the covariance matrix, given by LaMotte (1972), and avoids inversion of large matrices. Description of the algorithm concentrates on computational aspects for large sets of data. Computational methods for maximum likelihood estimation in unbalanced variance component models were developed by Hemmerle & Hartley (1973) using the W-transfor- mation, and by Patterson & Thompson (1971); see also Thompson (1980). These methods were reviewed by Harville (1977) who also discussed a variety of applications for the variance component models. Computational problems may arise when the number of clusters or random coefficients is large because inversion of very large matrices is required, and so there are severe limitations on the size of practical problems that can be handled. Goldstein (1986) and Aitkin & Longford (1986) present arguments for routine use of variance component models in educational context, but their arguments are applicable for a much wider range of problems including social surveys, longitudinal data, repeated measurements or experiments and multivariate analysis. The formulation of the general EM algorithm by Dempster, Laird & Rubin (1977) has led to development of alternative computational algorithms for variance component analysis by Dempster, Rubin & Tsutakawa (1981), Mason, Wong & Entwisle (1984) and others. These algorithms avoid inversion of large matrices, but may be very slow on complex problems, a common feature of EM algorithms. Convergence is especially slow when the variance components are small. The present paper gives details of a Fisher scoring algorithm for the unbalanced nested random effects model which converges rapidly and does not require the inversion of large matrices. The algorithm exploits the formulae for the inverse and the determinant of the irregularly patterned covariance matrix of the observations given by LaMotte (1972). The analysis presented by Aitkin & Longford (1986) uses software based on this algorithm. For another example see Longford (1985).

On the Iterative Image Space Reconstruction Algorthm for ECT

Abstract: The ISRA of [1] is shown to be an iterative algorithm that aims to converge to the least-squares estimates of emission densities. Convergence is established in the case where a unique least-squares estimate exists that is, elementwise, strictly positive. It is pointed out that, in terms of asymptotic theory, the resulting estimators are inferior to the maximum likelihood estimators, for which the EM algorithm is a computational procedure. Potential difficulties with the behavior of the ISRA are illustrated using very simple examples.

Mixed-model analysis of a censored normal distribution with reference to animal breeding.

Abstract: A mixed-model procedure for analysis of censored data assuming a multivariate normal distribution is described. A Bayesian framework is adopted which allows for estimation of fixed effects and variance components and prediction of random effects when records are left-censored. The procedure can be extended to right- and two-tailed censoring. The model employed is a generalized linear model, and the estimation equations resemble those arising in analysis of multivariate normal or categorical data with threshold models. Estimates of variance components are obtained using expressions similar to those employed in the EM algorithm for restricted maximum likelihood (REML) estimation under normality.

Exponential family non‐linear models for categorical data with errors of observation

Abstract: We present a general framework for treating categorical data with errors of observation. We show how both latent class models and models for doubly sampled data can be treated as exponential family nonlinear models. These are extended generalized linear models with the link function substituted by an observationwise defined non-linear function of the model parameters. The models are formulated in terms of structural probabilities and conditional error probabilities, thus allowing natural constraints when modelling errors of observation. We use an iteratively reweighted least squares procedure for obtaining maximum likelihood estimates. This is faster than the traditionally used EM algorithm and the computations can be made in GLIM.1 As examples we analyse three sets of categorical data with errors of observation which have been analysed before by Ashford and Sowden,2 Goodman3 and Chen,4 respectively.

Semiparametric estimation in the Rasch model

Abstract: A method for estimating the parameters of the Rasch model is examined. The unknown quantities in this method are the item parameters and the distribution function of the latent trait over the population. In this sense, the method is equivalent to marginal maximum likelihood estimation. The new procedure is based on a method suggested by J. Kiefer and J. Wolfowitz (1956). Their conclusions are reviewed, and links to the Rasch model are specified. In marginal maximum likelihood estimation, the item parameters are estimated first, and then the prior distribution of the person parameters is estimated using these estimates. The proposed method illustrates that it is possible to estimate these two quantities together and arrive at consistent estimates.

Maximum likelihood estimation via the alternating projection maximization algorithm

Abstract: We present a novel and efficient algorithm for computing the maximum likelihood estimator of the locations of multiple sources in passive sensor arrays. The algorithm is equally well applicable to the case of fully correlated signals appearing, for example, in multipath propagation problems. Simulation results that demonstrate the performance of the algorithm and a detailed analysis of the uniqueness of the solution are included.

Estimating the parameters of mixture models with modal estimators

Abstract: This paper extends some of the work presented in Redner and Walker [I9841 on the maximum likelihood estimate of parameters in a mixture model to a Bayesian modal estimate. The problem of determining the mode of the joint posterior distribution is discussed. Necessary conditions are given for a choice of parameters to be the mode and a numerical scheme based on the EM algorithm is presented. Some theoretical remarks on the resulting iterative scheme and simulation results are also given.

On maximum likelihood estimators for the multisite lag-one streamflow model: Complete and incomplete data cases

Abstract: Two practical problems may arise when using the traditional estimators of the parameters in the multisite lag-one streamflow model: (1) the estimated covariance matrix may not be positive definite thereby preventing its decomposition, a necessary step for synthetic streamflow generation, and (2) if there are missing observations, streamflow data must be truncated to the shortest record resulting in a loss of useful information. This note draws attention to the existence of maximum likelihood estimators which overcome both problems. In the complete data case, explicit maximum likelihood estimators exist. However, in the incomplete data case, iterative maximum likelihood procedures must be used. In particular, the expectation maximization (EM) algorithm is considered in a state-space framework compatible with the multisite streamflow model; this algorithm has robust convergence properties, is simple to implement, and produces smoothed estimates of the missing data. An example is presented illustrating problems with decomposition of the traditional covariance matrix estimate and also illustrating application of the EM algorithm.

An Incomplete Data Approach to the Analysis of Covariance Structures.

Abstract: In this paper, linear structural equation models with latent variables are considered. It is shown how many common models arise from incomplete observation of a relatively simple system. Subclasses of models with conditional independence interpretations are also discussed. Using an incomplete data point of view, the relationships between the incomplete and complete data likelihoods, assuming normality, are highlighted. For computing maximum likelihood estimates, the EM algorithm and alternatives are surveyed. For the alternative algorithms, simplified expressions for computing function values and derivatives are given. Likelihood ratio tests based on complete and incomplete data are related, and an example on using their relationship to improve the fit of a model is given.

Methods for noise cancellation based on the EM algorithm

Abstract: Single microphone speech enhancement systems have typically shown limited performance, while multiple microphone systems based on a least-squares error criterion have shown encouraging results in some contexts. In this paper we formulate a new approach to multiple microphone speech enhancement. Specifically, we formulate a maximum likelihood (ML) problem for estimating the parameters needed for canceling the noise in a two microphone speech enhancement system. This ML problem is solved via the iterative EM (Estimate-Maximize) technique. The resulting algorithm shows encouraging results when applied to the speech enhancement problem.

Statistical signal processing using a class of iterative estimation algorithms

Abstract: : Many Signal Processing problems may be posed as statistical parameter estimation problems A desired solution for the statistical problem is obtained by maximizing the Likelihood(ML), the A-Posteriori probability (MAP) or by optimizing other criterion, depending on the a-priori knowledge However, in many practical situations, the original signal processing problem may generate a complicated optimization problem eg when the observed signals are noisy and 'incomplete' A framework of iterative procedures for maximizing the likelihood, the EM algorithm, is widely used in statistics In the EM algorithm, the observations are considered 'incomplete' and the algorithm iterates between estimating the sufficient statistics of the 'complete data' given the observations and a current estimate of the parameters (the E step) and maximizing the likelihood of the complete data, using the estimated sufficient statistics (the M step) When this algorithm is applied to signal processing problems, it yields, in many cases, an intuitively appealing processing scheme

Maximum Likelihood Reconstruction for SPECT with Monte Carlo Modeling: Asymptotic Behavior

Abstract: Inverse Monte Carlo (IMOC) reconstruction for SPECT uses a Maximum Likelihood (EM) estimator with detection probabilities generated as Monte Carlo solutions to the photon transport equation (PTE). To evaluate the behavior of the iterative EM algorithm, reconstructions from experimental projection data for up to 1000 iterations were examined. Compensation for scatter and attenuation was achieved by including those effects in the PTE. For uniform activity distributions, noise increased monotonically with iteration. With line sources included, both noise and resolution improved between 1 and 30 iterations after which resolution was slightly improved at the expense of noise. Contrast (for a nonactive region surrounded by activity) improved from 0.32 at 5 iterations to 0.90 at 200 iterations, and 0.97 at 1000. Uncertainty in the measurement increased due to increased noise in the active region.

Segmentation of noisy textured images using simulated annealing

Abstract: This paper presents a segmentation algorithm for noisy textured images. To represent noisy textured images, we propose a hierarchical stochastic model that consists of three levels of random fields: the region process, the texture processes and the noise. The hierarchical model also includes local blurring and nonlinear image transformation as results of the image corrupting effects. Having adopted a statistical model, the maximum a posteriori (MAP) estimation is used to find the segmented regions through the restored(noise-free) textured image data. Since the joint a posteriori distribution at hand is a Gibbs distribution, we use simulated annealing as a maximization technique. The simulated annealing based segmentation algorithm presented in this paper can also be viewed as a two-step iterative algorithm in the spirit of the EM algorithm [10].

Maximum Likelihood Reconstruction for a Prototype Electronically Collimated Single Photon Emission System

Abstract: An electronic collimation system for SPECT imaging has been designed to provide improved detection efficiency over a mechanical collimator. A maximum likelihood estimator (MLE) is derived for this prototype electronically collimated system. The data is shown to be independent and Poisson distributed. For the low count rates typical in nuclear medicine the statistical fluctuations due to the Poisson process are significant. Consequently resolution in reconstructions using the linear formulations typical in x-ray computed tomography is limited by the resulting non-stationary noise. The MLE approach however incorporates the Poisson nature of the data directly in the reconstruction process and thus results in superior reconstructions than those obtained using a linear approach. Inclusion of noise effects in modeling the system is shown to guarantee the existence of a unique MLE solution. The EM algorithm is employed to find the MLE solution. The structure of the transition probability matrix obtained using a polar sampling raster is exploited to speed up the algorithm. Results of a comprehensive computer simulation are presented.

Maximum Likelihood Estimation of the Distribution of Length at Age

Abstract: SUMMARY This paper exploits the pattern of missing values in a multivariate otolith-fish length data set to obtain the maximum likelihood estimates of the age-length distribution. Methodological tools include maximum likelihood estimation, multiple regression, distance measures, and diagnostic graphical procedures. Data from a sample of cutthroat trout are used to illustrate the method and a simulation study is presented to compare alternative methods of estimation.

An efficient linear method for ARMA spectral estimation

Abstract: A three step method for obtaining nearly maximum likelihood ARMA spectral estimates is presented. The computational complexity of the algorithm is comparable to Yule-Walker methods, but the method gives asymptotically efficient estimates. The implementation of the algorithm is discussed, and numerical examples are presented to illustrate its performance.