Basics: Artificial Neurons

Affine Functions

Step #1. Weighted Sum :

$z = w_1x_1 + \dots + w_lx_l = (\overrightarrow{x})^T\overrightarrow{w}$

Step #2. Affine Transformation :

$z = w_1x_1 + \dots + w_lx_l + b = (\overrightarrow{x})^T\overrightarrow{w} + b$

Step #3. Affine Functions with n Features

$(\overrightarrow{x})^T = (x_1\; \dots \; x_l) \; \in \; \mathbb{R}^{1 \times l}$

$\overrightarrow{w} = (w_1\; \dots \; w_l) \; \in \; \mathbb{R}^{l \times 1}$

$b \; \in \; \mathbb{R}$

$f( \: (\overrightarrow{x})^T; \, \overrightarrow{w},\, b \: )$

$z \; \in \; \mathbb{R}$

Activation Functions

Sigmoid (logit -> probability)

$g(x) = \sigma(x) = {1 \over {1 + e^{-x}}}$

Tanh

$g(x) = tanh(x) = {{e^{x} - e^{-x}} \over {e^{x} + e^{-x}}}$

ReLU

$g(x) = ReLU(x) = max(0,\,x)$

$(\overrightarrow{z})^T = (x_1\; \dots \; x_l) \; \in \; \mathbb{R}^{1 \times l}$

$\overrightarrow{w} = (w_1\; \dots \; w_l) \; \in \; \mathbb{R}^{l \times 1}$

$b \; \in \; \mathbb{R}$

$f( \: (\overrightarrow{x})^T; \, \overrightarrow{w},\, b \: )$

$a \; \in \; \mathbb{R}$

Artificial Neurons

mini-batch in Artificial Neurons

$X^T \; \in \; \mathbb{R}^{N \times l}$

$neuron(x) = g( \: f( \: (\overrightarrow{x})^T; \, \overrightarrow{w},\, b \: ); \, \overrightarrow{w},\, b \: )$

$A \; \in \; \mathbb{R}^{N \times 1}$

$X^T \; \in \; \mathbb{R}^{N \times l}$

$\overrightarrow{w} = (w_1\; \dots \; w_l) \; \in \; \mathbb{R}^{l \times 1}$

$b \; \in \; \mathbb{R}$

$f( \: X^T; \, \overrightarrow{w},\, b \: )$

$Z \; \in \; \mathbb{R}^{N \times 1}$

$Z \; \in \; \mathbb{R}^{N \times 1}$

$g(Z)$

$A \; \in \; \mathbb{R}^{N \times 1}$