1 Newton-Rapshon for MLE

Define log likelihood as

\[\begin{equation} l(\boldsymbol{\theta}) = \sum_{i}^{n}\text{log}f(y_i;\boldsymbol{\theta}) \tag{1.1} \end{equation}\]

Score is gradient of log likelihood \(l(\boldsymbol{\theta})\)

\[\begin{equation} s(\boldsymbol{\theta}) = \frac{\partial l(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = \begin{bmatrix} \frac{l(\boldsymbol{\theta})}{\partial \theta_1}\\ \vdots\\ \frac{l(\boldsymbol{\theta})}{\partial \theta_n} \end{bmatrix} \tag{1.2} \end{equation}\]

Define Observed Information, or Hessian Matrix, as

\[\begin{equation} H(\boldsymbol{\theta}) = \frac{\partial^2 l(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial\boldsymbol{\theta}^T}=\frac{\partial s(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} \tag{1.3} \end{equation}\]

Assume that \(s(\boldsymbol{\theta}_{n+1}) = \mathbf{0}\), linear approximation of \(s(\boldsymbol{\theta}_{n+1}) = \mathbf{0}\) is

\[\begin{equation} s(\boldsymbol{\theta}_{n+1}) =\mathbf{0} \approx s(\boldsymbol{\theta}_{n}) + H(\boldsymbol{\theta}_{n})(\boldsymbol{\theta}_{n+1}-\boldsymbol{\theta}_{n}) \tag{1.4} \end{equation}\]

Equation (1.5) implies that

\[\begin{equation} \boldsymbol{\theta}_{n+1} = \boldsymbol{\theta}_{n} - H\left(\boldsymbol{\theta}_{n}\right)^{-1}s\left(\boldsymbol{\theta}_{n}\right) \tag{1.5} \end{equation}\]

2 Fisher scoring

Define Expected Information, as

\[\begin{equation} I(\boldsymbol{\theta})=\mathrm{E}(-H(\boldsymbol{\theta})) \tag{2.1} \end{equation}\]

Replace Observed Information with Expected Information, we have formula for Fisher Scoring

\[\begin{equation} \boldsymbol{\theta}_{n+1}=\boldsymbol{\theta}_{n}+I(\boldsymbol{\theta})^{-1} s\left(\boldsymbol{\theta}_{n}\right) \tag{2.2} \end{equation}\]

References

Agresti, Alan. 2016. Foundations of Linear and Generalized Linear Models. First Edition.
Nelder, J. A., and R. W. M. Wedderburn. 1972. “Generalized Linear Models.” Journal of the Royal Statistical Society. Series A (General) 135 (3): 370–84. http://www.jstor.org/stable/2344614.