Singularity and Slow Convergence of the EM algorithm for Gaussian Mixtures

2009 ◽  
Vol 29 (1) ◽  
pp. 45-59 ◽  
Author(s):  
Hyeyoung Park ◽  
Tomoko Ozeki
Keyword(s):  
Em Algorithm ◽  
Gaussian Mixtures ◽  
Slow Convergence ◽  
The Em Algorithm

Asymptotic Convergence Rate of the EM Algorithm for Gaussian Mixtures

2000 ◽  
Vol 12 (12) ◽  
pp. 2881-2907 ◽  
Author(s):  
Jinwen Ma ◽  
Lei Xu ◽  
Michael I. Jordan
Keyword(s):  
Convergence Rate ◽  
Em Algorithm ◽  
Iterative Algorithms ◽  
Asymptotic Convergence ◽  
Gaussian Mixtures ◽  
True Solution ◽  
First Order ◽  
The Em Algorithm ◽  
The Given ◽  
Asymptotic Convergence Rate

It is well known that the convergence rate of the expectation-maximization (EM) algorithm can be faster than those of convention first-order iterative algorithms when the overlap in the given mixture is small. But this argument has not been mathematically proved yet. This article studies this problem asymptotically in the setting of gaussian mixtures under the theoretical framework of Xu and Jordan (1996). It has been proved that the asymptotic convergence rate of the EM algorithm for gaussian mixtures locally around the true solution Θ* is o(e0.5−ε(Θ*)), where ε > 0 is an arbitrarily small number, o(x) means that it is a higher-order infinitesimal as x → 0, and e(Θ*) is a measure of the average overlap of gaussians in the mixture. In other words, the large sample local convergence rate for the EM algorithm tends to be asymptotically superlinear when e(Θ*) tends to zero.

scholarly journals On Convergence Properties of the EM Algorithm for Gaussian Mixtures

1996 ◽  
Vol 8 (1) ◽  
pp. 129-151 ◽  
Author(s):  
Lei Xu ◽  
Michael I. Jordan
Keyword(s):  
Em Algorithm ◽  
Gaussian Mixture Models ◽  
Gaussian Mixture ◽  
Gaussian Mixtures ◽  
Convergence Properties ◽  
Advantages And Disadvantages ◽  
The Em Algorithm ◽  
The Matrix ◽  
Gradient Based ◽  
Mathematical Connection

We build up the mathematical connection between the “Expectation-Maximization” (EM) algorithm and gradient-based approaches for maximum likelihood learning of finite gaussian mixtures. We show that the EM step in parameter space is obtained from the gradient via a projection matrix P, and we provide an explicit expression for the matrix. We then analyze the convergence of EM in terms of special properties of P and provide new results analyzing the effect that P has on the likelihood surface. Based on these mathematical results, we present a comparative discussion of the advantages and disadvantages of EM and other algorithms for the learning of gaussian mixture models.

Sign in / Sign up

Export Citation Format

Share Document