[ML] 2. Logistic Regression for Classification

Lecture Week 3. Logistic Regression for Classification

Logistic Regression
: a classification algorithm, used when the value of the target variable is categorical in nature. used when the data has binary output, belongs to one class or another, either a 0 or 1
$y \in \{0,1\}$

Mathematical Representation
$0 \leq h_\theta(x) \leq 1\\ \displaystyle h_\theta(x) = g(\theta^T x) = \frac{1}{1 + e^{\theta^T x}}$
$\displaystyle g(z) = \frac{1}{1 + e^{-z}}$: logistic function (sigmoid)
$\displaystyle P(y|x;\theta) = (h_\theta(x))^y(1-h_\theta(x))^{1-y}$ if y=0 or 1;1

Max Likelihood
x is independent, so the likelihood of all data = the product of the likelihood of each data
$\displaystyle L(\theta) = P(\overrightarrow{y}|X;\theta) = \prod^m_{i=1}P(y^i|x^i;\theta)$ : likelihood
$\displaystyle l(\theta) = logL(\theta) = \sum^m_{i=1}y^i log h(x^i) + (1 - y^i)log(1 - h(x^i))$ maximize the log likelihood until $l'(\theta)=0$
$\displaystyle \theta_j := \theta_j + \alpha\frac{\partial l(\theta)}{\partial\theta_j} = \theta_j + \alpha(y^i - h_\theta(x^i))x_j^i$ : gradient ascent

Min Cost Function
$cost(h_\theta(x), y) = -y(log(h_\theta(x))) - (1-y)log(1 - h_\theta(x))\\ = \begin{cases} -log(h_\theta(x)), \; if\; y=1\\ -log(1 - h_\theta(x)), \; if\; y=0 \end{cases}$

Newton's Method
: optimize $\theta$ such that $f'(\theta)=0$

$\displaystyle \theta := \theta - \frac{f(\theta)}{f'(\theta)}$

maximize $l(\theta)$ taking bigger steps, converge earlier than gradient ascent
let $f(\theta) = l'(\theta)\\ \displaystyle \theta^{(t+1)} = \theta^{(t)} - \frac{l'(\theta^{(t)})}{l''(\theta^{(t)})}$