[Linear Algebra] Another Derivation of Normal Equation

Another Derivation of Normal Equation

• $\hat{\textbf{x}} = \underset{\textbf{x}}{\textrm{argmin}}\begin{Vmatrix} \textbf{b} - A \textbf{x} \end{Vmatrix}$
  $= \underset{\textbf{x}}{\textrm{argmin}}\begin{Vmatrix} \textbf{b} - A \textbf{x} \end{Vmatrix}^2$
  $= \underset{\textbf{x}}{\textrm{argmin}}\begin{pmatrix} \textbf{b} - A \textbf{x} \end{pmatrix}^T \begin{pmatrix} \textbf{b} - A \textbf{x} \end{pmatrix}$
  $= \underset{\textbf{x}}{\textrm{argmin}}\begin{pmatrix} \textbf{b} \textbf{b}^T - \textbf{x}^T A^T \textbf{b} - \textbf{b}^T A \textbf{x} + \textbf{x}^T A^T A \textbf{x} \end{pmatrix}$

• Computing derivatives of $\begin{pmatrix} \textbf{b} \textbf{b}^T - \textbf{x}^T A^T \textbf{b} - \textbf{b}^T A \textbf{x} + \textbf{x}^T A^T A \textbf{x} \end{pmatrix}$ w.r.t. $\textbf{x}$,
  we obtain $- A^T \textbf{b} - A^T \textbf{b} + A^T A \textbf{x} + A^T A \textbf{x} = \textbf{0} \Leftrightarrow A^T A \textbf{x} = A^T \textbf{b}$

  • if $C = A^T A$ is invertible, then the solution is computed as $\textbf{x} = (A^T A)^{-1} A^T \textbf{b}$