Backpropagation

Forward Propagation

For this Multi layer perceptron(MLP) of a binary classification.
The enumerated forward propagation is like this.

• Forward propagation path

1. input $\textbf{x}$ // size (2,1)
2. linear combination in hidden layer $\textbf{z}^{[2]}$ // size (2,1)

3. non-linearity function $\textbf{a}^{[2]}$ // size (2,1)

   ${a}^{[2]} = \textrm{tanh}( \textbf{z}^{[2]} )$

4. linear combination in output layer $\textbf{z}^{[3]}$ // size (1, 1)

   $\textbf{z}^{[3]} = \textbf{W}^{[3]} \textbf{x}, (W= 2 \times 1 \:\:\text{matrix})$

5. sigmoid function for binary classification $\textbf{a}^{[3]}$ // size (1,1)

   $\hat{y} = {a}^{[3]} = \sigma( \textbf{z}^{[3]} )$

6. cross entrpy function $l$

   $l(\textbf{x}_i, \textbf{y}_i; \textbf{W}^{[2]}, \textbf{b}^{[2]}, \textbf{W}^{[3]}, \textbf{b}^{[3]}) = -(y_ilog\hat{y_i} + (1-y_i) log(1-\hat{y_i}))$

Backward propagation rule

single node

• node : linear combination or non-linearity happen

Donwstream gradient = Local gradient * Upstream gradient

Multiple node

• multiple inputs invoke multiple local gradients

computing - output layer

• In output layer we want to get the dervatives below
  $\frac{\partial{l}}{\partial{\textbf{a}}^{[3]}}$

• If there are $n_L$ (output) units in Layer $L$, then $\frac{\partial{l}}{\partial{\textbf{a}^L}}$ and $\frac{\partial{l}}{\partial{\textbf{z}^L}}$ are vectors with $n_L$ elements, $\frac{\partial{\textbf{a}^L}}{\partial{\textbf{z}^L}}$ is $n_L$ * $n_L$ Jacobian matrix:

If , $f_L$(non-linearity function) is applied element-wise(e.g., sigmoid) then this matrix is diagonal.

• because each non-relevant derivatives are zero except mathcing diagonal elements (e.g., $a_1 = \sigma(z_1)$, $a_2 = \sigma(z_2)$,..., $a_n = \sigma(z_n)$)

computing - hidden layer

• the inputs into layer $I+1$

• for this linear combination,
• upstream derivatives: $\frac{\partial{l}}{\partial{\textbf{z}}^{I+1}}$
• local derivatives: $\frac{\partial{\textbf{z}}^{I+1}}{\partial{\textbf{z}}^{I}}$
• downstream derivatives: $\frac{\partial{l}}{\partial{\textbf{z}}^{I}}$

computing - each parameter

• $\textbf{W}^I$

• upstream derivatives: $\frac{\partial{l}}{\partial{\textbf{z}^{l}}}$

• local derivatives: $\frac{\partial{\textbf{z}^I}}{\partial{\textbf{w}^{l}}}$

• downstream derivatives:

  $\frac{\partial{l}}{\partial{\textbf{w}^{l}}} = \frac{\partial{l}}{\partial{\textbf{z}^{l}}} \frac{\partial{\textbf{z}^I}}{\partial{\textbf{w}^{l}}}$
  $\frac{\partial{l}}{\partial{\textbf{w}^{l}}}= \textbf{a}^{l-1} \cdot \frac{\partial{l}}{\partial{\textbf{z}^{l}}}$

• $\textbf{b}^I$

• upstream derivatives: $\frac{\partial{l}}{\partial{\textbf{z}^{l}}}$

• local derivatives: $\frac{\partial{\textbf{z}^I}}{\partial{\textbf{b}^{l}}} = I$

• downstream derivatives:

  $\frac{\partial{l}}{\partial{\textbf{b}^{l}}}= \frac{\partial{l}}{\partial{\textbf{z}^{l}}}$

Backpropagation Fast calculation Tip

• Assume fc network looks like this
  

• Layer part including $\mathbf{z}$ and $\mathbf{a}$, as activation function

  $\mathbf{a}_l = f(\mathbf{z}_l)$

• And between Layer l-1 and Layer l, there are weight $\mathbf{W}_l$ and bias $\mathbf{b}_l$. Then from the output of Layer l-1, the equaiton as follows.

• We assumed that all the gradient of $l$ with respect $\mathbf{z}$ is already calculated. Such as..

  $\frac{\partial l}{\partial \mathbf{z}_2}, \frac{\partial l}{\partial \mathbf{z}_3}, \dotsc, \frac{\partial l}{\partial \mathbf{z}_l}, \dotsc, \frac{\partial l}{\partial \mathbf{z}_L}$

• In this situataion, if we want to get the gradient of weight in specific layer($l$).
  Just multiply inputted $\mathbf{a}_{l-1}$ and the gradient from outputted $\mathbf{z}_l$

• Bias is much more simple, it is just the gradient with respect to $\mathbf{z}_l$.
  $\frac{\partial{l}}{\partial{\textbf{b}^{l}}}= \frac{\partial{l}}{\partial{\textbf{z}^{l}}}$