Showing papers on "Expectation–maximization algorithm published in 2004"

A Tutorial on MM Algorithms

Abstract: Most problems in frequentist statistics involve optimization of a function such as a likelihood or a sum of squares. EM algorithms are among the most effective algorithms for maximum likelihood estimation because they consistently drive the likelihood uphill by maximizing a simple surrogate function for the log-likelihood. Iterative optimization of a surrogate function as exemplified by an EM algorithm does not necessarily require missing data. Indeed, every EM algorithm is a special case of the more general class of MM optimization algorithms, which typically exploit convexity rather than missing data in majorizing or minorizing an objective function. In our opinion, MM algorithms deserve to be part of the standard toolkit of professional statisticians. This article explains the principle behind MM algorithms, suggests some methods for constructing them, and discusses some of their attractive features. We include numerous examples throughout the article to illustrate the concepts described. In addition t...

Generalized Latent Variable Modeling: Multilevel, Longitudinal, and Structural Equation Models

Abstract: METHODOLOGY THE OMNI-PRESENCE OF LATENT VARIABLES Introduction 'True' variable measured with error Hypothetical constructs Unobserved heterogeneity Missing values and counterfactuals Latent responses Generating flexible distributions Combining information Summary MODELING DIFFERENT RESPONSE PROCESSES Introduction Generalized linear models Extensions of generalized linear models Latent response formulation Modeling durations or survival Summary and further reading CLASSICAL LATENT VARIABLE MODELS Introduction Multilevel regression models Factor models and item response models Latent class models Structural equation models with latent variables Longitudinal models Summary and further reading GENERAL MODEL FRAMEWORK Introduction Response model Structural model for the latent variables Distribution of the disturbances Parameter restrictions and fundamental parameters Reduced form of the latent variables and linear predictor Moment structure of the latent variables Marginal moment structure of observed and latent responses Reduced form distribution and likelihood Reduced form parameters Summary and further reading IDENTIFICATION AND EQUIVALENCE Introduction Identification Equivalence Summary and further reading ESTIMATION Introduction Maximum likelihood: Closed form marginal likelihood Maximum likelihood: Approximate marginal likelihood Maximizing the likelihood Nonparametric maximum likelihood estimation Restricted/Residual maximum likelihood (REML) Limited information methods Maximum quasi-likelihood Generalized Estimating Equations (GEE) Fixed effects methods Bayesian methods Summary Appendix: Some software and references ASSIGNING VALUES TO LATENT VARIABLES Introduction Posterior distributions Empirical Bayes (EB) Empirical Bayes modal (EBM) Maximum likelihood Relating the scoring methods in the 'linear case' Ad hoc scoring methods Some uses of latent scoring and classification Summary and further reading Appendix: Some software MODEL SPECIFICATION AND INFERENCE Introduction Statistical modeling Inference (likelihood based) Model selection: Relative fit criteria Model adequacy: Global absolute fit criteria Model diagnostics: Local absolute fit criteria Summary and further reading APPLICATIONS DICHOTOMOUS RESPONSES Introduction Respiratory infection in children: A random intercept model Diagnosis of myocardial infarction: A latent class model Arithmetic reasoning: Item response models Nicotine gum and smoking cessation: A meta-analysis Wives' employment transitions: Markov models with unobserved heterogeneity Counting snowshoe hares: Capture-recapture models with heterogeneity Attitudes to abortion: A multilevel item response model Summary and further reading ORDINAL RESPONSES Introduction Cluster randomized trial of sex education: Latent growth curve model Political efficacy: Factor dimensionality and item-bias Life satisfaction: Ordinal scaled probit factor models Summary and further reading COUNTS Introduction Prevention of faulty teeth in children: Modeling overdispersion Treatment of epilepsy: A random coefficient model Lip cancer in Scotland: Disease mapping Summary and further reading DURATIONS AND SURVIVAL Introduction Modeling multiple events clustered duration data Onset of smoking: Discrete time frailty models Exercise and angina: Proportional hazards random effects and factor models Summary and further reading COMPARATIVE RESPONSES Introduction Heterogeneity and 'Independence from Irrelevant Alternatives' Model structure British general elections: Multilevel models for discrete choice and rankings Post-materialism: A latent class model for rankings Consumer preferences for coffee makers: A conjoint choice model Summary and further reading MULTIPLE PROCESSES AND MIXED RESPONSES Introduction Diet and heart disease: A covariate measurement error model Herpes and cervical cancer: A latent class covariate measurement error model for a case-control study Job training and depression: A complier average causal effect model Physician advice and drinking: An endogenous treatment model Treatment of liver cirrhosis: A joint survival and marker model Summary and further reading REFERENCES INDEX AUTHOR INDEX

A study of smoothing methods for language models applied to information retrieval

Abstract: Language modeling approaches to information retrieval are attractive and promising because they connect the problem of retrieval with that of language model estimation, which has been studied extensively in other application areas such as speech recognition. The basic idea of these approaches is to estimate a language model for each document, and to then rank documents by the likelihood of the query according to the estimated language model. A central issue in language model estimation is smoothing, the problem of adjusting the maximum likelihood estimator to compensate for data sparseness. In this article, we study the problem of language model smoothing and its influence on retrieval performance. We examine the sensitivity of retrieval performance to the smoothing parameters and compare several popular smoothing methods on different test collections. Experimental results show that not only is the retrieval performance generally sensitive to the smoothing parameters, but also the sensitivity pattern is affected by the query type, with performance being more sensitive to smoothing for verbose queries than for keyword queries. Verbose queries also generally require more aggressive smoothing to achieve optimal performance. This suggests that smoothing plays two different role---to make the estimated document language model more accurate and to "explain" the noninformative words in the query. In order to decouple these two distinct roles of smoothing, we propose a two-stage smoothing strategy, which yields better sensitivity patterns and facilitates the setting of smoothing parameters automatically. We further propose methods for estimating the smoothing parameters automatically. Evaluation on five different databases and four types of queries indicates that the two-stage smoothing method with the proposed parameter estimation methods consistently gives retrieval performance that is close to---or better than---the best results achieved using a single smoothing method and exhaustive parameter search on the test data.

Item Response Theory: Parameter Estimation Techniques

Abstract: The Item Characteristic Curve: Dichotomous Response Estimating the Parameters of an Item Characteristic Curve Maximum Likelihood Estimation of Examinee Ability Maximum Likelihood Procedures for Estimating Both Ability and Item Parameters The Rasch Model Marginal Maximum Likelihood Estimation and an EM Algorithm Bayesian Parameter Estimation Procedures The Graded Item Response Nominally Scored Items Markov Chain Monte Carlo Methods Parameter Estimation with Multiple Groups Parameter Estimation for a Test with Mixed Item Types

Maximum likelihood estimation of endogenous switching regression models

Abstract: This article describes the movestay Stata command, which implements the maximum likelihood method to fit the endogenous switching regression model.

FlexMix: A general framework for finite mixture models and latent class regression in R

Abstract: FlexMix implements a general framework for fitting discrete mixtures of regression models in the R statistical computing environment: three variants of the EM algorithm can be used for parameter estimation, regressors and responses may be multivariate with arbitrary dimension, data may be grouped, e.g., to account for multiple observations per individual, the usual formula interface of the S language is used for convenient model specification, and a modular concept of driver functions allows to interface many dierent types of regression models. Existing drivers implement mixtures of standard linear models, generalized linear models and model-based clustering. FlexMix provides the E-step and all data handling, while the M-step can be supplied by the user to easily define new models.

A mixture model for clustering ensembles

Abstract: Clustering ensembles have emerged as a powerful method for improving both the robustness and the stability of unsupervised classification solutions. However, finding a consensus clustering from multiple partitions is a difficult problem that can be approached from graph-based, combinatorial or statistical perspectives. We offer a probabilistic model of consensus using a finite mixture of multinomial distributions in a space of clusterings. A combined partition is found as a solution to the corresponding maximum likelihood problem using the EM algorithm. The excellent scalability of this algorithm and comprehensible underlying model are particularly important for clustering of large datasets. This study compares the performance of the EM consensus algorithm with other fusion approaches for clustering ensembles. We also analyze clustering ensembles with incomplete information and the effect of missing cluster labels on the quality of overall consensus. Experimental results demonstrate the effectiveness of the proposed method on large real-world datasets. keywords: unsupervised learning, clustering ensemble, consensus function, mixture model, EM algorithm.

LSimpute: accurate estimation of missing values in microarray data with least squares methods.

Abstract: Microarray experiments generate data sets with information on the expression levels of thousands of genes in a set of biological samples. Unfortunately, such experiments often produce multiple missing expression values, normally due to various experimental problems. As many algorithms for gene expression analysis require a complete data matrix as input, the missing values have to be estimated in order to analyze the available data. Alternatively, genes and arrays can be removed until no missing values remain. However, for genes or arrays with only a small number of missing values, it is desirable to impute those values. For the subsequent analysis to be as informative as possible, it is essential that the estimates for the missing gene expression values are accurate. A small amount of badly estimated missing values in the data might be enough for clustering methods, such as hierachical clustering or K-means clustering, to produce misleading results. Thus, accurate methods for missing value estimation are needed. We present novel methods for estimation of missing values in microarray data sets that are based on the least squares principle, and that utilize correlations between both genes and arrays. For this set of methods, we use the common reference name LSimpute. We compare the estimation accuracy of our methods with the widely used KNNimpute on three complete data matrices from public data sets by randomly knocking out data (labeling as missing). From these tests, we conclude that our LSimpute methods produce estimates that consistently are more accurate than those obtained using KNNimpute. Additionally, we examine a more classic approach to missing value estimation based on expectation maximization (EM). We refer to our EM implementations as EMimpute, and the estimate errors using the EMimpute methods are compared with those our novel methods produce. The results indicate that on average, the estimates from our best performing LSimpute method are at least as accurate as those from the best EMimpute algorithm.

A new algorithm for haplotype‐based association analysis: the Stochastic‐EM algorithm

Abstract: Summary It is now widely accepted that haplotypic information can be of great interest for investigating the role of a candidate gene in the etiology of complex diseases. In the absence of family data, haplotypes cannot be deduced from genotypes, except for individuals who are homozygous at all loci or heterozygous at only one site. Statistical methodologies are therefore required for inferring haplotypes from genotypic data and testing their association with a phenotype of interest. Two maximum likelihood algorithms are often used in the context of haplotype-based association studies, the Newton-Raphson (NR) and the Expectation-Maximisation (EM) algorithms. In order to circumvent the limitations of both algorithms, including convergence to local minima and saddle points, we here described how a stochastic version of the EM algorithm, referred to as SEM, could be used for testing haplotypephenotype association. Statistical properties of the SEM algorithm were investigated through a simulation study for a large range of practical situations, including small/large samples and rare/frequent haplotypes, and results were compared to those obtained by use of the standard NR algorithm. Our simulation study indicated that the SEM algorithm provides results similar to those of the NR algorithm, making the SEM algorithm of great interest for haplotype-based association analysis, especially when the number of polymorphisms is quite large.

Cyclic minimizers, majorization techniques, and the expectation-maximization algorithm: a refresher

Abstract: Many parameter estimation problems in signal processing can be reduced to the task of minimizing a function of the unknown parameters. This task is difficult owing to the existence of possibly local minima and the sharpness of the global minimum. In this article we review three approaches that can be used to minimize functions of the type encountered in parameter estimation problems. The first two approaches, the cyclic minimization and the majorization technique, are quite general, whereas the third one, the expectation-maximization (EM) algorithm, is tied to the use of the maximum likelihood (ML) method for parameter estimation. The article provides a quick refresher of the aforementioned approaches for a wide readership.

Data streaming algorithms for efficient and accurate estimation of flow size distribution

Abstract: Knowing the distribution of the sizes of traffic flows passing through a network link helps a network operator to characterize network resource usage, infer traffic demands, detect traffic anomalies, and accommodate new traffic demands through better traffic engineering. Previous work on estimating the flow size distribution has been focused on making inferences from sampled network traffic. Its accuracy is limited by the (typically) low sampling rate required to make the sampling operation affordable. In this paper we present a novel data streaming algorithm to provide much more accurate estimates of flow distribution, using a "lossy data structure" which consists of an array of counters fitted well into SRAM. For each incoming packet, our algorithm only needs to increment one underlying counter, making the algorithm fast enough even for 40 Gbps (OC-768) links. The data structure is lossy in the sense that sizes of multiple flows may collide into the same counter. Our algorithm uses Bayesian statistical methods such as Expectation Maximization to infer the most likely flow size distribution that results in the observed counter values after collision. Evaluations of this algorithm on large Internet traces obtained from several sources (including a tier-1 ISP) demonstrate that it has very high measurement accuracy (within 2%). Our algorithm not only dramatically improves the accuracy of flow distribution measurement, but also contributes to the field of data streaming by formalizing an existing methodology and applying it to the context of estimating the flow-distribution.

Recursive unsupervised learning of finite mixture models

Abstract: There are two open problems when finite mixture densities are used to model multivariate data: the selection of the number of components and the initialization. In this paper, we propose an online (recursive) algorithm that estimates the parameters of the mixture and that simultaneously selects the number of components. The new algorithm starts with a large number of randomly initialized components. A prior is used as a bias for maximally structured models. A stochastic approximation recursive learning algorithm is proposed to search for the maximum a posteriori (MAP) solution and to discard the irrelevant components.

Iterative maximum-likelihood reconstruction in quantum homodyne tomography

Abstract: I propose an iterative expectation maximization algorithm for reconstructing the density matrix of an optical ensemble from a set of balanced homodyne measurements. The algorithm applies directly to the acquired data, bypassing the intermediate step of calculating marginal distributions. The advantages of the new method are made manifest by comparing it with the traditional inverse Radon transformation technique.

Shape Matching and Recognition – Using Generative Models and Informative Features

Abstract: We present an algorithm for shape matching and recognition based on a generative model for how one shape can be generated by the other This generative model allows for a class of transformations, such as affine and non-rigid transformations, and induces a similarity measure between shapes The matching process is formulated in the EM algorithm To have a fast algorithm and avoid local minima, we show how the EM algorithm can be approximated by using informative features, which have two key properties–invariant and representative They are also similar to the proposal probabilities used in DDMCMC [13] The formulation allows us to know when and why approximations can be made and justifies the use of bottom-up features, which are used in a wide range of vision problems This integrates generative models and feature-based approaches within the EM framework and helps clarifying the relationships between different algorithms for this problem such as shape contexts [3] and softassign [5] We test the algorithm on a variety of data sets including MPEG7 CE-Shape-1, Kimia silhouettes, and real images of street scenes We demonstrate very effective performance and compare our results with existing algorithms Finally, we briefly illustrate how our approach can be generalized to a wider range of problems including object detection

A variational approach for Bayesian blind image deconvolution

Abstract: In this paper, the blind image deconvolution (BID) problem is addressed using the Bayesian framework. In order to solve for the proposed Bayesian model, we present a new methodology based on a variational approximation, which has been recently introduced for several machine learning problems, and can be viewed as a generalization of the expectation maximization (EM) algorithm. This methodology reaps all the benefits of a "full Bayesian model" while bypassing some of its difficulties. We present three algorithms that solve the proposed Bayesian problem in closed form and can be implemented in the discrete Fourier domain. This makes them very cost effective even for very large images. We demonstrate with numerical experiments that these algorithms yield promising improvements as compared to previous BID algorithms. Furthermore, the proposed methodology is quite general with potential application to other Bayesian models for this and other imaging problems.

A Bayesian approach to joint feature selection and classifier design

Abstract: This paper adopts a Bayesian approach to simultaneously learn both an optimal nonlinear classifier and a subset of predictor variables (or features) that are most relevant to the classification task. The approach uses heavy-tailed priors to promote sparsity in the utilization of both basis functions and features; these priors act as regularizers for the likelihood function that rewards good classification on the training data. We derive an expectation- maximization (EM) algorithm to efficiently compute a maximum a posteriori (MAP) point estimate of the various parameters. The algorithm is an extension of recent state-of-the-art sparse Bayesian classifiers, which in turn can be seen as Bayesian counterparts of support vector machines. Experimental comparisons using kernel classifiers demonstrate both parsimonious feature selection and excellent classification accuracy on a range of synthetic and benchmark data sets.

Depth-variant maximum-likelihood restoration for three-dimensional fluorescence microscopy

Abstract: We derive an algorithm for maximum-likelihood image estimation on the basis of the expectation-maximization (EM) formalism by using a new approximate model for depth-varying image formation for optical sectioning microscopy. This new strata-based model incorporates spherical aberration that worsens as the microscope is focused deeper under the cover slip and is the result of the refractive-index mismatch between the immersion medium and the mounting medium of the specimen. Images of a specimen with known geometry and refractive index show that the model captures the main features of the image. We analyze the performance of the depth-variant EM algorithm with simulations, which show that the algorithm can compensate for image degradation changing with depth.

Using an EM Covariance Matrix to Estimate Structural Equation Models with Missing Data: Choosing an Adjusted Sample Size to Improve the Accuracy of Inferences.

Abstract: Two methods, direct maximum likelihood (ML) and the expectation maximization (EM) algorithm, can be used to obtain ML parameter estimates for structural equation models with missing data (MD). Although the 2 methods frequently produce identical parameter estimates, it may be easier to satisfy missing at random assumptions using EM. However, no single value of N is applicable to the EM covariance matrix, and this may compromise inferences gained from the model fit statistic and parameter standard errors. The purpose of this study was to identify a value of N that provides accurate inferences when using EM. If all confirmatory factor analysis model indicators have MD, results suggest that the minimum N per covariance term yields honest Type 1 error rates. If MD are restricted to a subset of indicators, the minimum N per variance works well. With respect to standard errors, the harmonic mean N per variance term produces honest confidence interval coverage rates.

On maximum likelihood estimation of the extreme value index

Abstract: We prove asymptotic normality of the so-called maximum likelihood estimator of the extreme value index.

Learning multiple evolutionary pathways from cross-sectional data

Abstract: We introduce a mixture model of trees to describe evolutionary processes that are characterized by the accumulation of permanent genetic changes. The basic building block of the model is a directed weighted tree that generates a probability distribution on the set of all patterns of genetic events. We present an EM-like algorithm for learning a mixture model of K trees and show how to determine K with a maximum likelihood approach. As a case study we consider the accumulation of mutations in the HIV-1 reverse transcriptase that are associated with drug resistance. The fitted model is statistically validated as a density estimator and the stability of the model topology is analyzed. We obtain a generative probabilistic model for the development of drug resistance in HIV that agrees with biological knowledge. Further applications and extensions of the model are discussed.

Statistical list-mode image reconstruction for the high resolution research tomograph

Abstract: We have investigated statistical list-mode reconstruction applicable to a depth-encoding high resolution research tomograph. An image non-negativity constraint has been employed in the reconstructions and is shown to effectively remove the overestimation bias introduced by the sinogram non-negativity constraint. We have furthermore implemented a convergent subsetized (CS) list-mode reconstruction algorithm, based on previous work (Hsiao et al 2002 Conf. Rec. SPIE Med. Imaging 4684 10-19; Hsiao et al 2002 Conf. Rec. IEEE Int. Symp. Biomed. Imaging 409-12) on convergent histogram OSEM reconstruction. We have demonstrated that the first step of the convergent algorithm is exactly equivalent (unlike the histogram-mode case) to the regular subsetized list-mode EM algorithm, while the second and final step takes the form of additive updates in image space. We have shown that in terms of contrast, noise as well as FWHM width behaviour, the CS algorithm is robust and does not result in limit cycles. A hybrid algorithm based on the ordinary and the convergent algorithms is also proposed, and is shown to combine the advantages of the two algorithms (i.e. it is able to reach a higher image quality in fewer iterations while maintaining the convergent behaviour), making the hybrid approach a good alternative to the ordinary subsetized list-mode EM algorithm.

Three validation metrics for automated probabilistic image segmentation of brain tumours

Abstract: The validity of brain tumour segmentation is an important issue in image processing because it has a direct impact on surgical planning. We examined the segmentation accuracy based on three two-sample validation metrics against the estimated composite latent gold standard, which was derived from several experts' manual segmentations by an EM algorithm. The distribution functions of the tumour and control pixel data were parametrically assumed to be a mixture of two beta distributions with different shape parameters. We estimated the corresponding receiver operating characteristic curve, Dice similarity coefficient, and mutual information, over all possible decision thresholds. Based on each validation metric, an optimal threshold was then computed via maximization. We illustrated these methods on MR imaging data from nine brain tumour cases of three different tumour types, each consisting of a large number of pixels. The automated segmentation yielded satisfactory accuracy with varied optimal thresholds. The performances of these validation metrics were also investigated via Monte Carlo simulation. Extensions of incorporating spatial correlation structures using a Markov random field model were considered.

Robust Analysis of Generalized Linear Mixed Models

Abstract: The method of maximum likelihood (ML) is widely used for analyzing generalized linear mixed models (GLMM's). A full maximum likelihood analysis requires numerical integration techniques for calculation of the log-likelihood, and to avoid the computational problems involving irreducibly high-dimensional integrals, several maximum likelihood algorithms have been proposed in the literature to estimate the model parameters by approximating the log-likelihood function. Although these likelihood algorithms are useful for fitting the GLMM's efficiently under strict model assumptions, they can be highly influenced by the presence of unusual data points. In this article, the author develops a technique for finding robust maximum likelihood (RML) estimates of the model parameters in GLMM's, which appears to be useful in downweighting the influential data points when estimating the parameters. The asymptotic properties of the robust estimators are investigated under some regularity conditions. Small simulations are ...

A likelihood-based constrained algorithm for multivariate normal mixture models

Abstract: It is well known that the log-likelihood function for samples coming from normal mixture distributions may present spurious maxima and singularities. For this reason here we reformulate some Hathaway’s results and we propose two constrained estimation procedures for multivariate normal mixture modelling according to the likelihood approach. Their perfomances are illustrated on the grounds of some numerical simulations based on the EM algorithm. A comparison between multivariate normal mixtures and the hot-deck approach in missing data imputation is also considered.

Semi-parametric estimation of the binormal ROC curve for a continuous diagnostic test.

Abstract: SUMMARY Not until recently has much attention been given to deriving maximum likelihood methods for estimating the intercept and slope parameters from a binormal ROC curve that assesses the accuracy of a continuous diagnostic test. We propose two new methods for estimating these parameters. The first method uses the profile likelihood and a simple algorithm to produce fully efficient estimates. The second method is based on a pseudo-maximum likelihood that can easily accommodate adjusting for covariates that could affect the accuracy of the continuous test.