In this repository, I'll introduce 2 methods for Gaussian Mixture Model (GMM) estimation - EM algorithm (expectation-maximization algorithm) and variational inference (variational Bayes).
To make you have a clear picture, I'll also give you mathematical descriptions, with several lines of code in Python.
GMM formula has summation (not multiplication) in distribution, and the log likelihood will then lead to complex expression in regular maximum likelihood estimation (MLE). These 2 methods will then address this concern by procedural iterative algorithms (which approximate the optimal solutions).
EM algorithm is an iterative method based on maximum likelihood framework.
For this reason, there might have several difficulties in specific cases. For instance, when some data (observation) is same as a mean value of element in GMM, it might have a singularity, in which the likelihood goes to infinity at σ = 0. (This won't occur in a single Gaussian distribution.)
When the number of data is insufficient, this method might also over-estimate the results.
The motivation to apply variational Bayesian (variational inference) is to mitigate this kind of over-estimating and over-fitting by Bayesian approach.
Note : See here for the caveat of likelihood approach.
With Scikit-Learn package in Python, you can also use functions for both EM algorithm (sklearn.mixture.GaussianMixture) and variational Bayesian (sklearn.mixture.BayesianGaussianMixture) in GMM.
However, here I'll show you implementation from scratch in Python with mathematical explanations. By knowing mathematical background and implementation, these ideas can also be applied to other distributions - such as, mixtures of Bernoulli distributions, hidden Markov Models (HMM), etc. (See here for applying EM algorithms in HMM and LDS.)