Maximum Likelihood Parameter Estimation and the CR bound

Consider a general log PDF that depends on $P$ parameters: $\theta$$= [\theta_1, \theta_2 \ldots \theta_P]^\prime$:

$$p({\bf x}) = p({\bf x};$$   $$\mbox{\boldmath$\theta$}$$$$).$$

The Fisher's information between any two parameters $\theta_1$ and $\theta_2$ is defined by

$$I_{\theta_2, \theta_2} = - {\rm E}\left\{ \frac{\partial^2 \log p... ...f x};\mbox{\boldmath$\theta$}) }{\partial \theta_1 \partial \theta_2} \right\}.$$ (17.13)

Collecting all these values into the matrix ${\bf I}($$\theta$$)$, we have the Fisher's information matrix. The Cramer-Rao lower bound states that the covariance matrix ${\bf C}$ of any joint unbiased estimator for the parameters $\theta$ is such that

$${\rm det}\left\{{\bf C}-{\bf I}^{-1}(\mbox{\boldmath $\theta$})\right\} > 0$$

. This effectively means that ${\bf I}^{-1}($$\theta$$)$ is the lower bound for the covariance of any unbiased estimator.

The inverse of the Fisher's information matrix is a good estimate of the parameter estimation error covariance and is useful for iterative optimization. Given a parameter estimate $\theta$$_n$, the new estimate is obtained as

$$\mbox{\boldmath$\theta$} _{n+1}= \mbox{\boldmath$\theta$} _n + I^{-1}( \mbox{\boldmath$\theta$} _n) \; \mbox{\boldmath$\delta$} ,$$ (17.14)

where

$\delta$$$= \left[ D(\theta_1) \; D(\theta_2) \ldots \right]^\prime$$

is the gradient vector formed from the first partial derivatives

$$D(\theta)\stackrel{\mbox{\tiny $\Delta$}}{=}\frac{\partial}{\partial \theta} \log p({\bf x}; \theta).$$

It is possible to optimize only subsets of the features as well. A feature pair $\theta_1, \theta_2$ is updated according to

$$\left[ \begin{array}{l} \theta_1 \theta_2 \end{array}\right]_{... ...^{-1} \; \left[ \begin{array}{l} D(\theta_1) D(\theta_2) \end{array}\right].$$ (17.15)