[Linear Algebra] Linear System, Identity Matrix

Linear System : Set of Equations

• A system of linear equations (or a linear system) is a collection of one or more linear equations involving the same variables

Linear System Example

• Suppose we have ((Feature A, B, C), y) dataset

Feature A	Feature B	Feature C	label
2	2	1	6
3	1	0	14
1	3	1	18

• What we have to do is just solving the linear system below

$2 x_1 + 2 x_2 + 1 x_3 = 6$
$3 x_1 + 1 x_2 + 0 x_3 = 14$
$1 x_1 + 3 x_2 + 1 x_3 = 18$

• The essential information of a linear system can be written compactly using a matrix like below

$A \textbf{x} = \textbf{b}$

where $A = \begin{bmatrix} 2 & 2 & 1 \\ 3 & 1 & 0 \\ 1 & 3 & 1 \\ \end{bmatrix}$, $\textbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}$, $\textbf{b} = \begin{bmatrix} 6 \\ 14 \\ 18 \end{bmatrix}$

From Multiple Equations to Single Matrix Equation

$2 x_1 + 2 x_2 + 1 x_3 = 6$
$3 x_1 + 1 x_2 + 0 x_3 = 14$
$1 x_1 + 3 x_2 + 1 x_3 = 18$

$\begin{bmatrix} 2 & 2 & 1 \\ 3 & 1 & 0 \\ 1 & 3 & 1 \\ \end{bmatrix}\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} = \begin{bmatrix} 6 \\ 14 \\ 18 \end{bmatrix}$

Identity Matrix

• Definition : an identity matrix is a square matrix whose diagonal entries are all 1's, and other entries are zeros

  • e.g. $I_3 = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{bmatrix}$

• Identity matrix $I_n$ preserves any vector $\textbf{x} \in \mathbb{R}^{n}$ after multiplying $\textbf{x}$ by $I_n$

  • $\forall \textbf{x} \in \mathbb{R}^{n}$, $I_n \textbf{x} = \textbf{x}$