Showing papers on "Expectation–maximization algorithm published in 1984"
••
TL;DR: This work discusses the formulation and theoretical and practical properties of the EM algorithm, a specialization to the mixture density context of a general algorithm used to approximate maximum-likelihood estimates for incomplete data problems.
Abstract: The problem of estimating the parameters which determine a mixture density has been the subject of a large, diverse body of literature spanning nearly ninety years. During the last two decades, the...
2,836 citations
••
TL;DR: In this paper, a necessary and sufficient condition for the consistency if the pseudo maximum likelihood estimation of the first and second moments is given is given, and the existence of a lower bound for the asymptotic covariance matrix is shown.
Abstract: Estimators obtained by maximizing a likelihood function are studied in the case where the true p.d.f. does not necessarily belong to the family chosen for the likelihood function. When such a procedure is applied to the estimation of the parameters of the first order moments, it is possible to prove a necessary and sufficient condition for its consistency. Asymptotic normality is shown as well as the existence of a lower bound for the asymptotic covariance matrix. It is also seen that this bound can be reached if consistent estimates are available for the parameters of the second order moments. Finally, a necessary and sufficient condition for the consistency if the pseudo maximum likelihood estimation of the first and second moments is given.
1,292 citations
••
TL;DR: In this paper, the maximum likelihood method for fitting the linear model when residuals are correlated and when the covariance among the residuals is determined by a parametric model containing unknown parameters is described.
Abstract: We describe the maximum likelihood method for fitting the linear model when residuals are correlated and when the covariance among the residuals is determined by a parametric model containing unknown parameters. Observations are assumed to be Gaussian. We give conditions which ensure consistency and asymptotic normality of the estimators. Our main concern is with the analysis of spatial data and in this context we describe some simulation experiments to assess the small sample behaviour of estimators. We also discuss an application of the spectral approximation to the likelihood for processes on a lattice.
858 citations
••
TL;DR: A general mixed model for the analysis of serial dichotomous responses provided by a panel of study participants, assuming each subject's serial responses are assumed to arise from a logistic model, but with regression coefficients that vary between subjects.
Abstract: This paper presents a general mixed model for the analysis of serial dichotomous responses provided by a panel of study participants. Each subject's serial responses are assumed to arise from a logistic model, but with regression coefficients that vary between subjects. The logistic regression parameters are assumed to be normally distributed in the population. Inference is based upon maximum likelihood estimation of fixed effects and variance components, and empirical Bayes estimation of random effects. Exact solutions are analytically and computationally infeasible, but an approximation based on the mode of the posterior distribution of the random parameters is proposed, and is implemented by means of the EM algorithm. This approximate method is compared with a simpler two-step method proposed by Korn and Whittemore (1979, Biometrics 35, 795-804), using data from a panel study of asthmatics originally described in that paper. One advantage of the estimation strategy described here is the ability to use all of the data, including that from subjects with insufficient data to permit fitting of a separate logistic regression model, as required by the Korn and Whittemore method. However, the new method is computationally intensive.
763 citations
••
TL;DR: In this article, the authors present maximum likelihood equations for the estimation of the population parameters directly from the observed responses; i.e., without estimating an ability parameter for each subject, and provide asymptotic standard errors and tests of fit, computing approximations, and details of four special cases: a nonparametric approximation, a normal solution, a resolution of normal components, and a beta-binomial solution.
Abstract: Consider vectors of item responses obtained from a sample of subjects from a population in which abilityθ is distributed with densityg(θ‖α), where theα are unknown parameters. Assuming the responses depend onθ through a fully specified item response model, this paper presents maximum likelihood equations for the estimation of the population parameters directly from the observed responses; i.e., without estimating an ability parameter for each subject. Also provided are asymptotic standard errors and tests of fit, computing approximations, and details of four special cases: a non-parametric approximation, a normal solution, a resolution of normal components, and a beta-binomial solution.
331 citations
••
TL;DR: In this paper, the authors consider binary regression models when some of the predictors are measured with error and show that if the measurement error is large, the usual estimate of the probability of the event in question can be substantially in error, especially for high risk groups.
Abstract: SUMMARY We consider binary regression models when some of the predictors are measured with error. For normal measurement errors, structural maximum likelihood estimates are considered. We show that if the measurement error is large, the usual estimate of the probability of the event in question can be substantially in error, especially for high risk groups. In the situation of large measurement error, we investigate a conditional maximum likelihood estimator and its properties.
215 citations
••
137 citations
••
TL;DR: In this article, a maximum likelihood procedure for the estimation of the parameters of the mover-stayer model is presented and a recursive method of computation of maximum likelihood estimators that is very simple to implement.
Abstract: The discrete time mover-stayer model, a special mixture of two independent Markov chains, has been widely used in modeling the dynamics of social processes. The problem of maximum likelihood estimation of its parameters from the data, however, which consist of a sample of independent realizations of this process, has not been considered in the literature. I present a maximum likelihood procedure for the estimation of the parameters of the mover-stayer model and develop a recursive method of computation of maximum likelihood estimators that is very simple to implement. I also verify that obtained maximum likelihood estimators are strongly consistent. I show that the two estimators of the parameters of the mover-stayer model previously proposed in the literature are special cases of the maximum likelihood estimator derived in this article, that is, they coincide with the maximum likelihood estimator under special conditions. I thus explain the interconnection between existing estimators. I also pre...
112 citations
••
TL;DR: In this paper, statistical and computational techniques for the analysis of data from a normal mixed model with two variances are discussed and illustrated, and two iterative algorithms for restricted maximum likelihood estimation (REML) of the variance components are compared.
Abstract: SUMMARY Statistical and computational techniques for the analysis of data from a normal mixed model with two variances are discussed and illustrated. Two iterative algorithms for restricted maximum likelihood estimation (REML) of the variances are compared. It is shown that these algorithms are much simplified by the use of a preliminary eigenvalue-eigenvector analysis. Two numerical examples are used to illustrate the theory by showing how variance estimates are used in the estimation and testing of fixed effects in the model. Monte Carlo simulations indicate that actual alpha levels of the tests are close to the nominal levels despite the estimation of the variance components. Diagnostic techniques are employed to assess model assumptions.
108 citations
••
TL;DR: In this article, the estimation of mixing proportions in the mixture model is discussed, with emphasis on the mixture of two normal components with all five parameters unknown, and simulations are presented that compare minimum distance (MD) and maximum likelihood (ML) estimation of the parameters of this mixture-of-normals model.
Abstract: The estimation of mixing proportions in the mixture model is discussed, with emphasis on the mixture of two normal components with all five parameters unknown. Simulations are presented that compare minimum distance (MD) and maximum likelihood (ML) estimation of the parameters of this mixture-of-normals model. Some practical issues of implementation of these results are also discussed. Simulation results indicate that ML techniques are superior to MD when component distributions actually are normal, but MD techniques provide better estimates than ML under symmetric departures from component normality. Interestingly, an ad hoc starting value for the iterative procedures occasionally outperformed both the ML and MD techniques. Results are presented that establish strong consistency and asymptotic normality of the MD estimator under conditions that include the mixture-of-normals model. Asymptotic variances and relative efficiencies are obtained for further comparison of the MD and ML estimators.
91 citations
••
TL;DR: In this article, it was shown that the exact finite sample distribution of the limited information maximum likelihood estimator in a general and leading single equation case is multivariate Cauchy.
Abstract: It is shown that the exact finite sample distribution of the limited information maximum likelihood (LIML) estimator in a general and leading single equation case is multivariate Cauchy. When the LIML estimator utilizes a known error covariance matrix (LIMLK) it is proved that the same Cauchy distribution still applies. The corresponding result for the instrumental variable (IV) estimator is a form of multivariate t density where the degrees of freedom depend on the number of instruments.(This abstract was borrowed from another version of this item.)
••
TL;DR: A general maximum likelihood algorthm, called the delta algorithm, is introduced, which generalizes Fisher's scoring method and several other existing algorithms, and leads to a general definition of residuals.
Abstract: Summary A general maximum likelihood algorthm, called the delta algorithm, which generalizes Fisher's scoring method and several other existing algorithms, is introduced. The algorithm is derived as a modification of the Newton-Raphson algorithm, and may be interpreted as an iterative weighted least squares method. We show that for certain models, the algorithm may be implemented in GuM, allowing a number of new models to be fitted in GuM. The algorithm is applied to marginal and conditional maximum likelihood estimation, and the relation with the EM algorithm for incomplete data problems is discussed. Finally, the approach leads to a general definition of residuals, which we consider in some detail.
••
••
TL;DR: Advantages and disadvantages of joint maximum likelihood, marginal maximum likelihood and Bayesian methods of parameter estimation in item response theory are discussed and compared in this article, where the authors compare the advantages of the three methods.
Abstract: Advantages and disadvantages of joint maximum likelihood, marginal maximum likelihood, and Bayesian methods of parameter estimation in item response theory are discussed and compared
••
TL;DR: In this article, the EM algorithm is used to derive maximum likelihood estimates of item parameters of the two-parameter logistic item response curves, which are then used to approximate the covariances of these estimates.
Abstract: : This paper presents a method for estimating certain characteristics of test items which are designed to measure ability, or knowledge, in a particular area. Under the assumption that ability parameters are sampled from a normal distribution, the EM algorithm is used to derive maximum likelihood estimates of item parameters of the two-parameter logistic item response curves. The observed information matrix is used to approximate the covariances of these estimates. Responses to a questionnaire on general arthritis knowledge are used to illustrate the procedure and simulated data are used to compare the actual versus estimated items parameters. A computational note is included to facilitate the extensive numerical work required to implement the procedure. (Author)
••
TL;DR: In this article, consistency and asymptotic normality of the m.l.i.d. case were examined in the non-i.i., i.e., when the parameters are constrained.
Abstract: Consistency and asymptotic normality of the m.l.e. are examined in the non-i.i.d. case when the parameters are constrained. Inequality constraints are considered as an application.
••
TL;DR: In this paper, the maximum likelihood estimates for the parameters of a mixture of two normal distributions are presented in terms of an expectation-maximization algorithm, and small sample properties of the parameter estimates are explored using Monte Carlo simulation.
Abstract: Maximum likelihood estimates for the parameters of a mixture of two normal distributions are presented in terms of an expectation-maximization algorithm. Small sample properties of the parameter estimates are explored using Monte Carlo simulation. Although parameters estimated from unclassified data are inaccurate, quantiles derived from the fitted distributions are only slightly less accurate than quantiles estimated from classified data.
••
TL;DR: In this paper, the distribution of the likelihood ratio statistic is considered for a number of systems of censoring and sequential stopping connected with Brownian motion, Poisson processes and survival analysis.
Abstract: Summary The distribution of the likelihood ratio statistic is considered for a number of systems of censoring and sequential stopping connected with Brownian motion, Poisson processes and survival analysis. The implications for the calculation of Bartlett adjustments are examined and the relation developed with an earlier general discussion of the higher-order distribution theory of maximum likelihood estimators and likelihood ratio statistics.
••
TL;DR: In this article, the authors extended the EM algorithm to a more general type of regression model, such as negative binomial or beta binomial distributions, and used it for censored and grouped data.
Abstract: Nelder and Wedderburn's method for maximum likelihood estimation of the parameters in an exponential family of regression models is extended to a more general type of model. Examples are given of the method's use for censored and grouped data, models involving the negative binomial or beta‐binomial distributions and in robust estimation. In a numerical example the algorithm converges considerably faster than the EM algorithm.
••
01 Jan 1984TL;DR: A number of methods for carrying out the maximum likelihood estimation of a dynamic econometric model with missing observations are examined and it is argued that in all cases the necessary computations can be carried out most efficiently by putting the model in state space form and applying the Kalman filter.
Abstract: A number of methods for carrying out the maximum likelihood estimation of a dynamic econometric model with missing observations are examined These include the approach suggested by Sargan and Drettakis and a method based on the EM algorithm The link between the different methods is explored and it is argued that in all cases the necessary computations can be carried out most efficiently by putting the model in state space form and applying the Kalman filter
••
TL;DR: In this paper, a truncated exponential family of absolutely continuous distributions with natural parameter θ and truncation parameter γ is considered and strong consistency and asymptotic normality are shown to hold for the maximum likelihood and maximum conditional likelihood estimates of θ with γ unknown.
Abstract: Consider a truncated exponential family of absolutely continuous distributions with natural parameter θ and truncation parameter γ. Strong consistency and asymptotic normality are shown to hold for the maximum likelihood and maximum conditional likelihood estimates of θ with γ unknown. Moreover, these two estimates are also shown to have the same limiting distribution, coinciding with that of the maximum likelihood estimate for θ when γ is assumed to be known.
••
TL;DR: In this article, the maximum likelihood equations for a multivariate normal model with structured mean and structured covariance matrix may not have an explicit solution, but within-iteration explicit solutions are shown for two general classes of models including covariance component models used for analysis of longitudinal data.
Abstract: The maximum likelihood equations for a multivariate normal model with structured mean and structured covariance matrix may not have an explicit solution. In some cases the model's error term may be decomposed as the sum of two independent error terms, each having a patterned covariance matrix, such that if one of the unobservable error terms is artificially treated as "missing data", the EM algorithm can be used to compute the maximum likelihood estimates for the original problem. Some decompositions produce likelihood equations which do not have an explicit solution at each iteration of the EM algorithm, but within-iteration explicit solutions are shown for two general classes of models including covariance component models used for analysis of longitudinal data.
••
TL;DR: Latent class analysis is formulated as a problem of estimating parameters in a finite mixture distribution using the EM algorithm to find the maximum likelihood estimates in categorical variables with more than two categories.
Abstract: Latent class analysis is formulated as a problem of estimating parameters in a finite mixture distribution. The EM algorithm is used to find the maximum likelihood estimates, and the case of categorical variables with more than two categories is considered.
••
01 Jan 1984TL;DR: The EM algorithm is reviewed here within the time series context and applied to the parameter estimation and smoothing problem for missing data state-space models and linear estimation (deconvolution) in a frequency domain regression model.
Abstract: One may encounter incompletely specified time series data in several distinct forms: (1) observations in time or space may be irregularly observed or (2) the underlying time series model may be incompletely observed, as in the case where one observes only the sum of a signal and a noise process. Maximum likelihood estimators for parameters in these missing data problems can be developed in a simple, heuristically appealing form by utilizing the EM (expectation-maximization) algorithm proposed by Dempster, et al. (1977) and others. Furthermore, the conditional expectations computed as a by-product of applying the algorithm are the empirical Bayes (in the sense of Efron and Morris (1973), (1975)) estimators for the unobserved components. The EM algorithm is reviewed here within the time series context and applied to (i) the parameter estimation and smoothing problem for missing data state-space models and (ii) linear estimation (deconvolution) in a frequency domain regression model.
••
TL;DR: In this article, an alternative technique for constructing a prediction function for the normal linear regression model based on the concept of maximum likelihood is proposed, and the form of this prediction function is evaluated and normalized to produce a multivariate Student's t-density.
Abstract: An alternative technique to current methods for constructing a prediction function for the normal linear regression model is proposed based on the concept of maximum likelihood. The form of this prediction function is evaluated and normalized to produce a multivariate Student's t-density. Consistency properties are established under regularity conditions, and an empirical comparison, based on the Kullback-Leibler information divergence, is made with some other prediction functions.
••
01 Jun 1984
TL;DR: An algorithm for maximum likelihood (ML) estimation is developed primarily for multivariable dynamic systems based on a new optimization method referred to as a modified Newton-Raphson with estimated sensitivities (MNRES), which eliminates the need to derive sensitivity equations.
Abstract: An algorithm for maximum likelihood (ML) estimation is developed primarily for multivariable dynamic systems. The algorithm relies on a new optimization method referred to as a modified Newton-Raphson with estimated sensitivities (MNRES). The method determines sensitivities by using slope information from local surface approximations of each output variable in parameter space. The fitted surface allows sensitivity information to be updated at each iteration with a significant reduction in computational effort compared with integrating the analytically determined sensitivity equations or using a finite-difference method. Different surface-fitting methods are discussed and demonstrated. Aircraft estimation problems are solved by using both simulated and real-flight data to compare MNRES with commonly used methods; in these solutions MNRES is found to be equally accurate and substantially faster. MNRES eliminates the need to derive sensitivity equations, thus producing a more generally applicable algorithm.
••
06 Jun 1984TL;DR: In this paper, a method for constrained maximum likelihood identification of the initial covariance of an otherwise known linear discrete time dynamical system, with guaranteed positive semi-definite estimate at each step, is presented.
Abstract: A method is presented for constrained maximum likelihood identification of the (positive semi-definite) initial covariance of an otherwise known linear discrete time dynamical system, with guaranteed positive semi-definite estimate at each step. The technique is a modification of Newton-Raphson or Scoring procedures transformed linearly to the space of the Cholesky square root matrix. The required algorithm is specified completely, and numerical and analytic difficulties and their solutions are discussed. It is shown that in cases of interest this procedure can result in order of magnitude reduction in computational costs compared to other iterative ML schemes which guarantee a semi-definite covariance estimate at each step. Formal extension to the maximum likelihood identification of time constants and power spectral densities is presented.
••
••
TL;DR: The EM (Estimation-Maximization) algorithm is exploited to provide maximum likelihood estimates of the parameters of multiple indicator/factor analysis models to reduce considerably the storage and computational burden of such estimation.
Abstract: The EM (Estimation-Maximization) algorithm is exploited to provide maximum likelihood estimates of the parameters of multiple indicator/factor analysis models. This method reduces considerably the storage and computational burden of such estimation. A computer program in BASIC language that performs the computations is listed in an appendix. The specification of correlated errors is also provided for in this application of the method.