Estimation using Gaussian Mixtures

The GM representation of the density has the a remarkable property that $p({\bf x}\vert{\bf y})$ can be computed in closed form. This is especially useful in visualization of information. For example, it is useful to show a human operator the distribution of likely ${\bf x}$ after ${\bf y}$ is measured. If desired, the MMSE can be computed in closed form as well. The MAP estimate can also be computed, but that requires a search over ${\bf x}$.

Let the GM approximation to the distribution be given by

$$p({\bf x},{\bf y})={ \displaystyle \sum}_{i}\alpha_{i}p_{i}({\bf x},{\bf y}).$$ (13.4)

By Bayes rule,

$$p({\bf x}\vert{\bf y})=\frac{p({\bf x},{\bf y})}{p({\bf y})} = \frac{1}{p({\bf y})} { \displaystyle \sum}_{i}\alpha_{i}p_{i}({\bf x},{\bf y})$$

where $p({\bf y})$ is the marginal distribution of ${\bf y}$. We now define $p_{i}({\bf y})$ as the marginal distributions of ${\bf y}$ given that ${\bf y}$ is a member of mode $i$. These are, of course, Gaussian with means and covariances taken from the ${\bf y}$-partitions of the mode $i$ mean and covariance $\mu$$_{i},{\bf\Sigma}_{i}$.

   $$\mbox{\boldmath$\mu$}$$$$_{i} = \left[ \begin{array}{l} \mbox{\boldmath$\mu$}_{x,i} \m... ...}_{xy,i} \\ {\bf\Sigma}_{yx,i} & {\bf\Sigma}_{yy,i} \\ \end{array} \right]$$

Then,

$$\begin{array}{rcl} p({\bf x}\vert{\bf y}) & = &{\displaystyle... ...alpha_{i} p_{i}({\bf y})p_{i}({\bf x}\vert{\bf y})} \end{array}$$ (13.5)

where $p_{i}({\bf x}\vert{\bf y})$ is the conditional density for ${\bf x}$ given ${\bf y}$ assuming that ${\bf x}$ and ${\bf y}$ are from that certain Gaussian sub-class $i$. Fortunately, there is a closed-form equation for $p_{i}({\bf x}\vert{\bf y})$ [63]. $p_{i}({\bf x}\vert{\bf y})$ is Gaussian with mean

$${\bf E}_{i}({\bf x}\vert{\bf y}) = \mbox{\boldmath$\mu$} _{x,i}+ {\bf\Sigma}_{xy,i} {\bf\Sigma}_{yy,i}^{-1} ({\bf y}- \mbox{\boldmath$\mu$} _{y,i}).$$ (13.6)

and covariance

$${\rm cov}_{i}({\bf x}\vert{\bf y}) = {\bf\Sigma}_{xx,i} - {\bf\Sigma}_{xy,i} {\bf\Sigma}_{yy,i}^{-1} {\bf\Sigma}_{yx,i}.$$ (13.7)

Note that the conditional distribution is a Gaussian Mixture in its own right, with mode weights modified by $p_{i}({\bf y})$ which tends to “select" the modes closest to ${\bf y}$. To reduce the number of modes in the conditioning process, one could easily remove those modes with a low value of $p_{i}({\bf y})$ (suggested by R. L. Streit).

This conditional distribution can be used for data visualization or, to easily calculate the conditional mean estimate, which is a by-product of equations (13.5),(13.6),(13.7):

$$\begin{array}{rcl} {\bf E}({\bf x}\vert{\bf y}) & = & {\displ... ...{i} p_{i}({\bf y}){\bf E}_{i}({\bf x}\vert{\bf y})} \end{array}$$ (13.8)