Maximum Likelihood Parameter Estimation and the CR bound
Consider a general log PDF that depends on parameters:
:
The Fisher's information between any two parameters
and is defined by

(16.13) 
Collecting all these values into the matrix
,
we have the Fisher's information matrix. The CramerRao lower bound states
that the covariance matrix of any joint unbiased estimator
for the parameters
is such that
. This
effectively means that
is the lower bound for
the covariance of any unbiased estimator.
The inverse of the Fisher's information matrix is a good estimate
of the parameter estimation error covariance and is useful for
iterative optimization. Given a parameter estimate
, the new estimate
is obtained as
where
is the gradient vector formed from the first partial derivatives
It is possible to optimize only subsets of the features as well. A feature pair
is updated according to

(16.15) 
Baggenstoss
20170519