[Linear Algebra] Least Squares Problem

Least Squares Problem

• Sum of squared errors : $\begin{Vmatrix} \textbf{b} - A\textbf{x} \end{Vmatrix}$
• Definition : given an overdetermined system $A\textbf{x} \simeq \textbf{b}$ where $A \in \mathbb{R}^{m \times n}, \textbf{x} \in \mathbb{R}^{m}, \textbf{b} \in \mathbb{R}^{n}$ and $m \gg n$, a lest squares solution $\hat{\textbf{x}}$ is defined as $\hat{\textbf{x}} = \underset{\textbf{x}}{\textrm{argmin}} \begin{Vmatrix} \textbf{b} - A\textbf{x} \end{Vmatrix}$
• The most important aspect of the least-squares problem is that no matter what $\textbf{x}$ we select, the vector $A \textbf{x}$ will necessarily be in the column space $\textrm{Col} A$
• Thus, we seek for $\textbf{x}$ that makes $A \textbf{x}$ as the closest point in $\textrm{Col} A$ to $\textbf{b}$